Sep 06 19:25:32.951984 ip-10-0-141-42 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 06 19:25:32.951996 ip-10-0-141-42 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 06 19:25:32.952006 ip-10-0-141-42 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 06 19:25:32.952834 ip-10-0-141-42 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 06 19:25:43.121609 ip-10-0-141-42 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 06 19:25:43.121628 ip-10-0-141-42 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3a5ece71af914deba84527e2fd7913b2 --
Sep 06 19:27:14.357191 ip-10-0-141-42 systemd[1]: Starting Kubernetes Kubelet...
Sep 06 19:27:14.774224 ip-10-0-141-42 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 19:27:14.774224 ip-10-0-141-42 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 06 19:27:14.774224 ip-10-0-141-42 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 19:27:14.774224 ip-10-0-141-42 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 06 19:27:14.774224 ip-10-0-141-42 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 19:27:14.775654 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.775564    2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 06 19:27:14.780041 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780023    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 19:27:14.780041 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780041    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780045    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780048    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780051    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780054    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780057    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780059    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780062    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780065    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780067    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780070    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780073    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780075    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780078    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780081    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780083    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780086    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780089    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780092    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780097    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 19:27:14.780105 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780100    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780102    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780105    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780107    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780110    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780113    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780115    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780118    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780121    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780123    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780126    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780128    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780131    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780133    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780136    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780138    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780141    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780144    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780147    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780149    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 19:27:14.780585 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780152    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780154    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780157    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780159    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780162    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780164    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780167    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780169    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780172    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780174    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780176    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780179    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780182    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780186    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780192    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780195    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780198    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780201    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780204    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 19:27:14.781080 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780206    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780209    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780212    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780214    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780217    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780219    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780222    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780224    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780226    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780229    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780233    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780236    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780238    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780241    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780243    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780246    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780248    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780252    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780256    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 19:27:14.781544 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780259    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780263    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780265    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780268    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780271    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780273    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780276    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780279    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780283    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780286    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780288    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780291    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780327    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780330    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780334    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780336    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780340    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780344    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780348    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.780351    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 19:27:14.782024 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781046    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781052    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781055    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781058    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781060    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781064    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781066    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781069    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781072    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781074    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781077    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781080    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781083    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781085    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781088    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781091    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781094    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781096    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781099    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 19:27:14.782550 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781102    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781106    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781109    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781111    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781114    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781116    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781119    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781122    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781124    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781127    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781129    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781132    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781135    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781137    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781140    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781142    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781145    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781148    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781150    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781153    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 19:27:14.783047 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781156    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781158    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781161    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781164    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781167    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781169    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781172    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781174    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781177    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781179    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781182    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781184    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781187    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781190    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781194    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781197    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781203    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781207    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781210    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781213    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 19:27:14.783538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781216    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781219    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781221    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781224    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781227    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781230    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781232    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781235    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781238    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781241    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781243    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781246    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781250    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781254    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781257    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781260    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781263    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781266    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781269    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 19:27:14.784056 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781272    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781275    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781279    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781282    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781285    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781288    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781292    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781295    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781298    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781301    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781304    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781306    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781309    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781312    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781314    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781317    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781319    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781322    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781324    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781327    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 19:27:14.784528 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781329    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781727    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781732    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781735    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781738    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781741    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781744    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781746    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781749    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781752    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781755    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781757    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781760    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781762    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781765    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781767    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781770    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781772    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781776    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781780    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 19:27:14.785025 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781783    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781786    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781788    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781791    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781794    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781796    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781799    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781801    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781804    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781806    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781809    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781811    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781813    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781816    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781818    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781821    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781823    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781826    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781829    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781845    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 19:27:14.785517 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781849    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781851    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781854    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781856    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781859    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781862    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781865    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781868    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781871    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781875    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781878    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781881    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781884    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781887    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781890    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781892    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781895    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781915    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781918    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781922    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 19:27:14.786031 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781925    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781928    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781931    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781934    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781937    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781939    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781942    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781944    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781947    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781949    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781952    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781955    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781958    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781960    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781963    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781965    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781980    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781985    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781989    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 19:27:14.786510 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781993    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781996    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.781999    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782004    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782008    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782011    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782017    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782020    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782023    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782026    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782029    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782031    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782034    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782037    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782039    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782042    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782045    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782047    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782050    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 19:27:14.787640 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782053    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782055    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782126    2571 flags.go:64] FLAG: --address="0.0.0.0"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782159    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782166    2571 flags.go:64] FLAG: --anonymous-auth="true"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782171    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782175    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782179    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782183    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782187    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782190    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782196    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782199    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782202    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782205    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782208    2571 flags.go:64] FLAG: --cgroup-root=""
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782212    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782215    2571 flags.go:64] FLAG: --client-ca-file=""
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782218    2571 flags.go:64] FLAG: --cloud-provider="external"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782222    2571 flags.go:64] FLAG: --cluster-dns="[]"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782227    2571 flags.go:64] FLAG: --cluster-domain=""
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782230    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782234    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782237    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 06 19:27:14.788162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782241    2571 flags.go:64] FLAG: --container-log-max-files="5"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782244    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782248    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782251    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782254    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782257    2571 flags.go:64] FLAG: --contention-profiling="false"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782260    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782263    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782266    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782269    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782273    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782276    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782279    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782282    2571 flags.go:64] FLAG: --enable-load-reader="false"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782285    2571 flags.go:64] FLAG: --enable-server="true"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782288    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782292    2571 flags.go:64] FLAG: --event-burst="100"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782296    2571 flags.go:64] FLAG: --event-qps="50"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782299    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782303    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782306    2571 flags.go:64] FLAG: --eviction-hard=""
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782309    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782312    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782315    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782318    2571 flags.go:64] FLAG: --eviction-soft=""
Sep 06 19:27:14.788744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782321    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782325    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782329    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782332    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782335    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782338    2571 flags.go:64] FLAG: --fail-swap-on="true"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782341    2571 flags.go:64] FLAG: --feature-gates=""
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782345    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782348    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782351    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782354    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782357    2571 flags.go:64] FLAG: --healthz-port="10248"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782360    2571 flags.go:64] FLAG: --help="false"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782363    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-141-42.ec2.internal"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782366    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782369    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782372    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782375    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782378    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782381    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782384    2571 flags.go:64] FLAG: --image-service-endpoint=""
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782387    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782390    2571 flags.go:64] FLAG: --kube-api-burst="100"
Sep 06 19:27:14.789353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782393    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782396    2571 flags.go:64] FLAG: --kube-api-qps="50"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782399    2571 flags.go:64] FLAG: --kube-reserved=""
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782403    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782406    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782410    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782413    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782416    2571 flags.go:64] FLAG: --lock-file=""
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782419    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782421    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782424    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782431    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782434    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782437    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782440    2571 flags.go:64] FLAG: --logging-format="text"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782443    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782446    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782449    2571 flags.go:64] FLAG: --manifest-url=""
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782452    2571 flags.go:64] FLAG: --manifest-url-header=""
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782456    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782459    2571 flags.go:64] FLAG: --max-open-files="1000000"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782463    2571 flags.go:64] FLAG: --max-pods="110"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782466    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782469    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782472    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 06 19:27:14.789963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782475    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782478    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782481    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782484    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782491    2571 flags.go:64] FLAG: --node-status-max-images="50"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782494    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782497    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782501    2571 flags.go:64] FLAG: --pod-cidr=""
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782503    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782509    2571 flags.go:64] FLAG: --pod-manifest-path=""
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782512    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782516    2571 flags.go:64] FLAG: --pods-per-core="0"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782520    2571 flags.go:64] FLAG: --port="10250"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782523    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782525    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09c4c25465be8a166"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782529    2571 flags.go:64] FLAG: --qos-reserved=""
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782531    2571 flags.go:64] FLAG: --read-only-port="10255"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782534    2571 flags.go:64] FLAG: --register-node="true"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782540    2571 flags.go:64] FLAG: --register-with-taints=""
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782544    2571 flags.go:64] FLAG: --registry-burst="10"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782547    2571 flags.go:64] FLAG: --registry-qps="5"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782549    2571 flags.go:64] FLAG: --reserved-cpus=""
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782552    2571 flags.go:64] FLAG: --reserved-memory=""
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782557    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782560    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 06 19:27:14.790604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782564    2571 flags.go:64] FLAG: --rotate-certificates="false"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782566    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782569    2571 flags.go:64] FLAG: --runonce="false"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782572    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782575    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782578    2571 flags.go:64] FLAG: --seccomp-default="false"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782581    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782584    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782587    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782591    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782594    2571 flags.go:64] FLAG: --storage-driver-password="root"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782596    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782599    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782602    2571 flags.go:64] FLAG: --storage-driver-user="root"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782605    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782608    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782611    2571 flags.go:64] FLAG: --system-cgroups=""
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782614    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782620    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782623    2571 flags.go:64] FLAG: --tls-cert-file=""
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782626    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782630    2571 flags.go:64] FLAG: --tls-min-version=""
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782633    2571 flags.go:64] FLAG: --tls-private-key-file=""
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782636    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782639    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 06 19:27:14.791588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782643    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782646    2571 flags.go:64] FLAG: --v="2"
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782649    2571 flags.go:64] FLAG: --version="false"
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782653    2571 flags.go:64] FLAG: --vmodule=""
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782657    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.782660    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782772    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782776    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782780    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782782    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782785    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782788    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782791    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782794    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782796    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782799    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782802    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782804    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782807    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782809    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782812    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 19:27:14.792506 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782814    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782817    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782819    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782822    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782826    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782829    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782847    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782850    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782853    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782855    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782858    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782861    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782864    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782867    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782870    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782872    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782875    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782878    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782881    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 19:27:14.793055 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782883    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782886    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782888    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782891    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782894    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782896    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782899    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782901    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782904    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782906    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782909    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782911    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782915    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782918    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782922    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782925    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782928    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782932    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782935    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782938    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 19:27:14.793529 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782941    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782943    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782946    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782949    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782951    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782954    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782956    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782959    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782963    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782966    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782970    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782974    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782977    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782980    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782983    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782986    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782988    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782991    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782993    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 19:27:14.794194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782996    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.782998    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783001    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783004    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783007    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783009    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783012    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783015    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783018    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783020    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783024    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783027    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783029    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783032    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783034    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783037    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783039    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783042    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783044    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783047    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 19:27:14.794657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783050    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 19:27:14.795161 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783052    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 19:27:14.795161 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783055    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 19:27:14.795161 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783058    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 19:27:14.795161 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783061    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 19:27:14.795161 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.783063    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 19:27:14.795161 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.783574    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 19:27:14.795161 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.792680    2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 06 19:27:14.795161 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.792696    2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 06 19:27:14.795161 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.793286    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 06 19:27:14.795161 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.793298    2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 06 19:27:14.795161 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793356    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 19:27:14.795161 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793361    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 19:27:14.795161 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793364    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 19:27:14.795161 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793367    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 19:27:14.795161 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793370    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793373    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793376    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793378    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793381    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793385    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793387    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793390    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793393    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793396    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793399    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793402    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793405    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793408    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793411    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793413    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793416    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793419    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793422    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793424    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 19:27:14.795527 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793427    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793430    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793433    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793438    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793441    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793443    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793446    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793449    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793452    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793455    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793457    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793460    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793463    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793465    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793468    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793470    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793473    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793475    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793478    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 19:27:14.796068 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793480    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793484    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793486    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793489    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793492    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793495    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793497    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793500    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793502    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793505    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793507    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793510    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793512    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793515    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793517    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793520    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793523    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793526    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793528    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793531    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 19:27:14.796538 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793534    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793536    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793539    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793542    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793546    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793549    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793552    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793555    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793559    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793563    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793566    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793569    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793572    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793575    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793578    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793580    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793584    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793587    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793589    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 19:27:14.797038 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793592    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793595    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793597    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793600    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793602    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793605    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793608    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793610    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793613    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793616    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793618    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793621    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793623    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793632    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793635    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793638    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 19:27:14.797568 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793641    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 19:27:14.797998 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.793646    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 19:27:14.797998 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793762    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 19:27:14.797998 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793767    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 19:27:14.797998 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793770    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 19:27:14.797998 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793773    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 19:27:14.797998 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793775    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 19:27:14.797998 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793778    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 19:27:14.797998 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793781    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 19:27:14.797998 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793783    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 19:27:14.797998 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793787    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 19:27:14.797998 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793790    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 19:27:14.797998 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793793    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 19:27:14.797998 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793795    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 19:27:14.797998 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793799    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 19:27:14.797998 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793801    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793804    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793806    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793809    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793812    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793814    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793817    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793819    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793823    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793827    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793846    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793852    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793855    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793858    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793860    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793863    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793866    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793869    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793871    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 19:27:14.798374 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793874    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793877    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793879    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793882    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793884    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793887    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793889    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793892    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793895    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793897    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793900    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793903    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793905    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793909    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793913    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793917    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793920    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793923    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793925    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 19:27:14.798938 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793928    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793931    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793934    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793936    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793939    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793942    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793944    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793947    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793949    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793952    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793954    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793957    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793960    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793962    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793965    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793967    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793969    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793972    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793975    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793977    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 19:27:14.799406 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793980    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793982    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793985    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793987    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793990    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793992    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793995    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.793999    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794001    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794004    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794006    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794009    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794011    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794014    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794016    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794019    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794021    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794023    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794026    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794028    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 19:27:14.799904 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794031    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 19:27:14.800416 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794033    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 19:27:14.800416 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794036    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 19:27:14.800416 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794038    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 19:27:14.800416 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794041    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 19:27:14.800416 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794044    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 19:27:14.800416 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794046    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 19:27:14.800416 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:14.794049    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 19:27:14.800416 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.794054    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 19:27:14.800416 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.794170    2571 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 06 19:27:14.800416 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.797070    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 06 19:27:14.800416 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.797859    2571 server.go:1019] "Starting client certificate rotation"
Sep 06 19:27:14.800416 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.797956    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 06 19:27:14.800416 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.798552    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 06 19:27:14.800745 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.800324    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 06 19:27:14.805437 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.805412    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 06 19:27:14.814884 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.814870    2571 log.go:25] "Validated CRI v1 runtime API"
Sep 06 19:27:14.817856 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.817843    2571 log.go:25] "Validated CRI v1 image API"
Sep 06 19:27:14.819447 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.819429    2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 06 19:27:14.823893 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.823871    2571 fs.go:135] Filesystem UUIDs: map[455f61a1-3640-4cfa-9ae2-d25766e1c2aa:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 abf92a0e-b35b-4d2e-bc58-35ba1d69735a:/dev/nvme0n1p3]
Sep 06 19:27:14.823963 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.823894    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 06 19:27:14.824120 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.824103    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 06 19:27:14.829537 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.829420    2571 manager.go:216] Machine: {Timestamp:2026-09-06 19:27:14.82807379 +0000 UTC m=+0.348496528 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100290 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eff6b336d4cccf952842593f0a840 SystemUUID:ec2eff6b-336d-4ccc-f952-842593f0a840 BootID:3a5ece71-af91-4deb-a845-27e2fd7913b2 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e9:1a:f1:58:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e9:1a:f1:58:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:8a:20:35:f9:4f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 06 19:27:14.829537 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.829528    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 06 19:27:14.829687 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.829625    2571 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 06 19:27:14.830995 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.830972    2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 06 19:27:14.831163 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.830997    2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-42.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 06 19:27:14.832163 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.832152    2571 topology_manager.go:138] "Creating topology manager with none policy"
Sep 06 19:27:14.832197 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.832166    2571 container_manager_linux.go:309] "Creating device plugin manager"
Sep 06 19:27:14.832197 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.832175    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 06 19:27:14.832958 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.832947    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 06 19:27:14.834195 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.834184    2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 06 19:27:14.838213 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.838200    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 06 19:27:14.838336 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.838326    2571 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 06 19:27:14.838373 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.838355    2571 kubelet.go:486] "Attempting to sync node with API server"
Sep 06 19:27:14.838373 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.838371    2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 06 19:27:14.838454 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.838386    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 06 19:27:14.838454 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.838419    2571 kubelet.go:398] "Adding apiserver pod source"
Sep 06 19:27:14.838454 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.838447    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 06 19:27:14.840017 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.839990    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 06 19:27:14.840109 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.840029    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 06 19:27:14.842333 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.842310    2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 06 19:27:14.844586 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.844573    2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 06 19:27:14.844637 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.844602    2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 06 19:27:14.844795 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.844783    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 06 19:27:14.844852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.844806    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 06 19:27:14.844852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.844816    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 06 19:27:14.844852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.844825    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 06 19:27:14.844852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.844850    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 06 19:27:14.844973 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.844860    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 06 19:27:14.845280 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.845270    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 06 19:27:14.845308 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.845281    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 06 19:27:14.845308 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.845289    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 06 19:27:14.845308 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.845295    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 06 19:27:14.845308 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.845304    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 06 19:27:14.845411 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.845312    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 06 19:27:14.846078 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.846068    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 06 19:27:14.846078 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.846079    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 06 19:27:14.850151 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.850137    2571 server.go:1268] "Started kubelet"
Sep 06 19:27:14.850237 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.850200    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 06 19:27:14.850295 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.850224    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 06 19:27:14.850295 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.850287    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 06 19:27:14.850712 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.850541    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-42.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 06 19:27:14.850779 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:14.850679    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-42.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 06 19:27:14.850824 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:14.850806    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 06 19:27:14.851245 ip-10-0-141-42 systemd[1]: Started Kubernetes Kubelet.
Sep 06 19:27:14.851500 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.851484    2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 06 19:27:14.852192 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.852179    2571 server.go:310] "Adding debug handlers to kubelet server"
Sep 06 19:27:14.858110 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:14.857159    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-42.ec2.internal.18d2d1f0f4decc6b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-42.ec2.internal,UID:ip-10-0-141-42.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-42.ec2.internal,},FirstTimestamp:2026-09-06 19:27:14.850106475 +0000 UTC m=+0.370529208,LastTimestamp:2026-09-06 19:27:14.850106475 +0000 UTC m=+0.370529208,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-42.ec2.internal,}"
Sep 06 19:27:14.859727 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.859708    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 06 19:27:14.859727 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.859710    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 06 19:27:14.859899 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:14.859845    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-42.ec2.internal\" not found"
Sep 06 19:27:14.860422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.860405    2571 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 06 19:27:14.860422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.860424    2571 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 06 19:27:14.860559 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.860546    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 06 19:27:14.860640 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:14.860606    2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 06 19:27:14.860700 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.860653    2571 reconstruct.go:97] "Volume reconstruction finished"
Sep 06 19:27:14.860700 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.860661    2571 reconciler.go:29] "Reconciler: start to sync state"
Sep 06 19:27:14.860785 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.860741    2571 factory.go:55] Registering systemd factory
Sep 06 19:27:14.860785 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.860758    2571 factory.go:223] Registration of the systemd container factory successfully
Sep 06 19:27:14.861011 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.860994    2571 factory.go:153] Registering CRI-O factory
Sep 06 19:27:14.861076 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.861015    2571 factory.go:223] Registration of the crio container factory successfully
Sep 06 19:27:14.861111 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.861082    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 06 19:27:14.861162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.861124    2571 factory.go:103] Registering Raw factory
Sep 06 19:27:14.861162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.861140    2571 manager.go:1196] Started watching for new ooms in manager
Sep 06 19:27:14.861621 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.861604    2571 manager.go:349] Starting recovery of all containers
Sep 06 19:27:14.862626 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:14.862571    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 06 19:27:14.862814 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:14.862790    2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-42.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 06 19:27:14.872545 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.872514    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z6z88"
Sep 06 19:27:14.872545 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.872534    2571 manager.go:354] Recovery completed
Sep 06 19:27:14.877187 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.876985    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z6z88"
Sep 06 19:27:14.878724 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.878711    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 19:27:14.880551 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.880527    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 19:27:14.880633 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.880566    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 19:27:14.880633 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.880582    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasSufficientPID"
Sep 06 19:27:14.881184 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.881170    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 06 19:27:14.881184 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.881183    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 06 19:27:14.881287 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.881198    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 06 19:27:14.881287 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.881232    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 19:27:14.881951 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.881937    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 19:27:14.882021 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.881965    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 19:27:14.882021 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.881978    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasSufficientPID"
Sep 06 19:27:14.883103 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:14.883018    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-42.ec2.internal.18d2d1f0f6af5095  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-42.ec2.internal,UID:ip-10-0-141-42.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-42.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-42.ec2.internal,},FirstTimestamp:2026-09-06 19:27:14.880549013 +0000 UTC m=+0.400971754,LastTimestamp:2026-09-06 19:27:14.880549013 +0000 UTC m=+0.400971754,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-42.ec2.internal,}"
Sep 06 19:27:14.883281 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.883265    2571 policy_none.go:49] "None policy: Start"
Sep 06 19:27:14.883281 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.883282    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 06 19:27:14.883405 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.883292    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 06 19:27:14.885288 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.885276    2571 policy_none.go:47] "Start"
Sep 06 19:27:14.928605 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.928585    2571 manager.go:337] "Starting Device Plugin manager"
Sep 06 19:27:14.938050 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:14.928638    2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 06 19:27:14.938050 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.928652    2571 server.go:87] "Starting device plugin registration server"
Sep 06 19:27:14.938050 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.929112    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 06 19:27:14.938050 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.929125    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 06 19:27:14.938050 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.929248    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 06 19:27:14.938050 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.929357    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 06 19:27:14.938050 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.929366    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 06 19:27:14.938050 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:14.930145    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 06 19:27:14.938050 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:14.930198    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-42.ec2.internal\" not found"
Sep 06 19:27:14.960130 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.960090    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 06 19:27:14.961365 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.961345    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 06 19:27:14.961486 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.961393    2571 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 06 19:27:14.961486 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.961417    2571 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 06 19:27:14.961486 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:14.961461    2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 06 19:27:14.963445 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:14.963426    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 19:27:15.030241 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.030137    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 19:27:15.031517 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.031499    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 19:27:15.031616 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.031530    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 19:27:15.031616 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.031543    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasSufficientPID"
Sep 06 19:27:15.031616 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.031576    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.040361 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.040340    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.040491 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:15.040369    2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-42.ec2.internal\": node \"ip-10-0-141-42.ec2.internal\" not found"
Sep 06 19:27:15.061696 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.061650    2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-42.ec2.internal"]
Sep 06 19:27:15.061802 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.061737    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 19:27:15.063472 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:15.063439    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-42.ec2.internal\" not found"
Sep 06 19:27:15.063753 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.063739    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 19:27:15.063812 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.063773    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 19:27:15.063812 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.063787    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasSufficientPID"
Sep 06 19:27:15.066179 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.066167    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 19:27:15.066311 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.066292    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.066359 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.066323    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 19:27:15.066995 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.066970    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 19:27:15.067152 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.067002    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 19:27:15.067152 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.067014    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 19:27:15.067152 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.067039    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 19:27:15.067152 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.067054    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasSufficientPID"
Sep 06 19:27:15.067152 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.067019    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasSufficientPID"
Sep 06 19:27:15.069245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.069229    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.069324 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.069253    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 19:27:15.069964 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.069947    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 19:27:15.070050 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.069968    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 19:27:15.070050 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.069977    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeHasSufficientPID"
Sep 06 19:27:15.092647 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:15.092617    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-42.ec2.internal\" not found" node="ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.097262 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:15.097243    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-42.ec2.internal\" not found" node="ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.161747 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.161712    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bee41a806707e9c28c0a2c777078070a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal\" (UID: \"bee41a806707e9c28c0a2c777078070a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.161747 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.161745    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bee41a806707e9c28c0a2c777078070a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal\" (UID: \"bee41a806707e9c28c0a2c777078070a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.161979 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.161764    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46e2ca63eb83f4b4b12734788288736a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-42.ec2.internal\" (UID: \"46e2ca63eb83f4b4b12734788288736a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.163655 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:15.163622    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-42.ec2.internal\" not found"
Sep 06 19:27:15.262826 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.262794    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bee41a806707e9c28c0a2c777078070a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal\" (UID: \"bee41a806707e9c28c0a2c777078070a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.262826 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.262829    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bee41a806707e9c28c0a2c777078070a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal\" (UID: \"bee41a806707e9c28c0a2c777078070a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.263054 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.262878    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46e2ca63eb83f4b4b12734788288736a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-42.ec2.internal\" (UID: \"46e2ca63eb83f4b4b12734788288736a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.263054 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.262902    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46e2ca63eb83f4b4b12734788288736a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-42.ec2.internal\" (UID: \"46e2ca63eb83f4b4b12734788288736a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.263054 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.262902    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bee41a806707e9c28c0a2c777078070a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal\" (UID: \"bee41a806707e9c28c0a2c777078070a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.263054 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.262964    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bee41a806707e9c28c0a2c777078070a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal\" (UID: \"bee41a806707e9c28c0a2c777078070a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.263824 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:15.263803    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-42.ec2.internal\" not found"
Sep 06 19:27:15.364647 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:15.364571    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-42.ec2.internal\" not found"
Sep 06 19:27:15.395767 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.395742    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal"
Sep 06 19:27:15.400705 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:15.400457    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-42.ec2.internal"
Sep 06 19:27:17.415578 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:17.415536    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-42.ec2.internal\" not found"
Sep 06 19:27:17.516009 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:17.515980    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-42.ec2.internal\" not found"
Sep 06 19:27:17.616493 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:17.616406    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-42.ec2.internal\" not found"
Sep 06 19:27:17.716913 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:17.716882    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-42.ec2.internal\" not found"
Sep 06 19:27:17.727434 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:17.727408    2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 19:27:17.748343 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:17.748315    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 06 19:27:17.748482 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:17.748444    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 06 19:27:17.748482 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:17.748471    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 06 19:27:17.817122 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:17.817090    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-42.ec2.internal\" not found"
Sep 06 19:27:17.830386 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:17.830350    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-05 19:22:16 +0000 UTC" deadline="2028-04-03 08:50:11.998529092 +0000 UTC"
Sep 06 19:27:17.830386 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:17.830381    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13789h22m54.168151734s"
Sep 06 19:27:17.918082 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:17.918020    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-42.ec2.internal\" not found"
Sep 06 19:27:17.927901 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:17.927879    2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 19:27:17.971622 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:17.971522    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e2ca63eb83f4b4b12734788288736a.slice/crio-e72b30b9eb4b47cfa7e168562f7088841c4e0707bc8fecf1059c464b913b8038 WatchSource:0}: Error finding container e72b30b9eb4b47cfa7e168562f7088841c4e0707bc8fecf1059c464b913b8038: Status 404 returned error can't find the container with id e72b30b9eb4b47cfa7e168562f7088841c4e0707bc8fecf1059c464b913b8038
Sep 06 19:27:17.972015 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:17.971983    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee41a806707e9c28c0a2c777078070a.slice/crio-ed59c1362c112b5aa4f3417d57d78413aaacf5765dd484e2faaa1ef17f6342a1 WatchSource:0}: Error finding container ed59c1362c112b5aa4f3417d57d78413aaacf5765dd484e2faaa1ef17f6342a1: Status 404 returned error can't find the container with id ed59c1362c112b5aa4f3417d57d78413aaacf5765dd484e2faaa1ef17f6342a1
Sep 06 19:27:17.978262 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:17.978098    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 19:27:18.010914 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.010871    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-42.ec2.internal"
Sep 06 19:27:18.018992 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.018972    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 06 19:27:18.019948 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.019925    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal"
Sep 06 19:27:18.029436 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.029410    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 06 19:27:18.375408 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.375324    2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 19:27:18.790431 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.790351    2571 apiserver.go:52] "Watching apiserver"
Sep 06 19:27:18.797331 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.797297    2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 06 19:27:18.797726 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.797703    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-5b6qn","openshift-ovn-kubernetes/ovnkube-node-n58nm","kube-system/global-pull-secret-syncer-mp9fn","kube-system/konnectivity-agent-rkkmb","kube-system/kube-apiserver-proxy-ip-10-0-141-42.ec2.internal","openshift-cluster-node-tuning-operator/tuned-tknwp","openshift-multus/multus-additional-cni-plugins-xp729","openshift-multus/network-metrics-daemon-pghst","openshift-network-operator/iptables-alerter-dmtzl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv","openshift-dns/node-resolver-fs4k9","openshift-image-registry/node-ca-z9s22","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal","openshift-multus/multus-qrvg8"]
Sep 06 19:27:18.800341 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.800318    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.802614 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.802571    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:18.802777 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:18.802657    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:18.802777 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.802717    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 06 19:27:18.802882 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.802793    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 06 19:27:18.802882 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.802795    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ssxfn\""
Sep 06 19:27:18.803042 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.803025    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 06 19:27:18.803112 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.803042    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 06 19:27:18.803112 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.803068    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 06 19:27:18.805009 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.804988    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:18.805110 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:18.805062    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:18.807271 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.807251    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rkkmb"
Sep 06 19:27:18.809372 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.809321    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 06 19:27:18.809372 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.809324    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pfx8g\""
Sep 06 19:27:18.809372 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.809350    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 06 19:27:18.810122 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.809454    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:18.810122 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:18.809513    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:18.810514 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.810498    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 06 19:27:18.811643 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.811624    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.813609 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.813590    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 06 19:27:18.813968 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.813941    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dmtzl"
Sep 06 19:27:18.814265 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.814243    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c7jfw\""
Sep 06 19:27:18.814489 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.814470    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 06 19:27:18.816568 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.816392    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.816853 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.816702    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 06 19:27:18.816853 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.816755    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q8fg6\""
Sep 06 19:27:18.816853 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.816801    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 06 19:27:18.817059 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.816935    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 06 19:27:18.818606 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.818587    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fm62m\""
Sep 06 19:27:18.818703 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.818611    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.819997 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.819975    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 06 19:27:18.821101 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.821080    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 06 19:27:18.821217 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.821194    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 06 19:27:18.821583 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.821565    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 06 19:27:18.821666 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.821645    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 06 19:27:18.821748 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.821655    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:18.821804 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.821755    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sdgt\" (UniqueName: \"kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt\") pod \"network-check-target-5b6qn\" (UID: \"11680e74-f55f-4749-a0a2-1180c6595524\") " pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:18.821861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.821787    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-sysctl-d\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.821861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.821836    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-host\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.821948 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.821883    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs\") pod \"network-metrics-daemon-pghst\" (UID: \"abf4350f-c25e-43ae-9271-5987625590f7\") " pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:18.821948 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.821938    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/349a592c-1bd1-461a-912a-25932cf842cc-agent-certs\") pod \"konnectivity-agent-rkkmb\" (UID: \"349a592c-1bd1-461a-912a-25932cf842cc\") " pod="kube-system/konnectivity-agent-rkkmb"
Sep 06 19:27:18.822060 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822000    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/349a592c-1bd1-461a-912a-25932cf842cc-konnectivity-ca\") pod \"konnectivity-agent-rkkmb\" (UID: \"349a592c-1bd1-461a-912a-25932cf842cc\") " pod="kube-system/konnectivity-agent-rkkmb"
Sep 06 19:27:18.822099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822056    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-kubernetes\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.822099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822086    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-lib-modules\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.822188 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822117    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.822230 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822178    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-var-lib-kubelet\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.822230 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822220    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-systemd\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.822316 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822248    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-tmp\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.822316 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822279    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z64dh\" (UniqueName: \"kubernetes.io/projected/abf4350f-c25e-43ae-9271-5987625590f7-kube-api-access-z64dh\") pod \"network-metrics-daemon-pghst\" (UID: \"abf4350f-c25e-43ae-9271-5987625590f7\") " pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:18.822316 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822310    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e5e01f7a-8ff8-4821-b662-aed4b0d11377-dbus\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:18.822473 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822370    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z6bq\" (UniqueName: \"kubernetes.io/projected/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-kube-api-access-9z6bq\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.822473 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822398    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-system-cni-dir\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.822473 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822438    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-cni-binary-copy\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.822473 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822468    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-modprobe-d\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.822654 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822493    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-run\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.822654 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822575    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-tuned\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.822654 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822630    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 06 19:27:18.822654 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822640    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 06 19:27:18.822843 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822655    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-cnibin\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.822843 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822764    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:18.822938 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822873    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kj5rk\""
Sep 06 19:27:18.823008 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822982    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 06 19:27:18.823816 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.822809    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-sysconfig\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.823910 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.823842    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-sysctl-conf\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.823910 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.823864    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-sys\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.824272 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.824244    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.824368 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.824283    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.824368 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.824312    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n56n\" (UniqueName: \"kubernetes.io/projected/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-kube-api-access-4n56n\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.824368 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.824338    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e5e01f7a-8ff8-4821-b662-aed4b0d11377-kubelet-config\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:18.824491 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.824481    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fs4k9"
Sep 06 19:27:18.826012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.825886    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 06 19:27:18.826012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.825915    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bzjtn\""
Sep 06 19:27:18.826148 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.826129    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 06 19:27:18.826380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.826363    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 06 19:27:18.827015 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.826993    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z9s22"
Sep 06 19:27:18.827109 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.827067    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 06 19:27:18.827167 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.827142    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 06 19:27:18.827231 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.827020    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w9q45\""
Sep 06 19:27:18.829364 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.829347    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 06 19:27:18.829617 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.829602    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 06 19:27:18.830103 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.830085    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 06 19:27:18.830262 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.830243    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k54l6\""
Sep 06 19:27:18.849284 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.849264    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-22whh"
Sep 06 19:27:18.855775 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.855755    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-22whh"
Sep 06 19:27:18.912077 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.911845    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 06 19:27:18.917055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.917002    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-42.ec2.internal" event={"ID":"46e2ca63eb83f4b4b12734788288736a","Type":"ContainerStarted","Data":"e72b30b9eb4b47cfa7e168562f7088841c4e0707bc8fecf1059c464b913b8038"}
Sep 06 19:27:18.918622 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.918592    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal" event={"ID":"bee41a806707e9c28c0a2c777078070a","Type":"ContainerStarted","Data":"ed59c1362c112b5aa4f3417d57d78413aaacf5765dd484e2faaa1ef17f6342a1"}
Sep 06 19:27:18.925467 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.925429    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-multus-conf-dir\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.925791 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.925768    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-run-openvswitch\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.925791 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.925797    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-log-socket\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.925984 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.925819    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.925984 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.925887    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-systemd\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.925984 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.925932    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-tmp\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.925984 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.925972    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z64dh\" (UniqueName: \"kubernetes.io/projected/abf4350f-c25e-43ae-9271-5987625590f7-kube-api-access-z64dh\") pod \"network-metrics-daemon-pghst\" (UID: \"abf4350f-c25e-43ae-9271-5987625590f7\") " pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:18.926147 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.925984    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-systemd\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.926147 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.925996    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e5e01f7a-8ff8-4821-b662-aed4b0d11377-dbus\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:18.926147 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926031    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-multus-socket-dir-parent\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.926147 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926058    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-kubelet\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.926147 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926083    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-systemd-units\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.926147 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926105    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-etc-openvswitch\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.926147 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926129    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z6bq\" (UniqueName: \"kubernetes.io/projected/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-kube-api-access-9z6bq\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.926403 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926150    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e5e01f7a-8ff8-4821-b662-aed4b0d11377-dbus\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:18.926403 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926152    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-var-lib-cni-bin\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.926403 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926217    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-socket-dir\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:18.926403 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926295    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-system-cni-dir\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.926403 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926338    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-var-lib-kubelet\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.926403 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926361    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-system-cni-dir\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.926595 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926413    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-modprobe-d\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.926595 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926443    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-tuned\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.926595 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926501    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-cnibin\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.926595 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926516    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-modprobe-d\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.926595 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926555    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:18.926749 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926603    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-cnibin\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.926749 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:18.926668    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 19:27:18.926749 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926669    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 06 19:27:18.926749 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926679    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:18.926749 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926709    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztgm9\" (UniqueName: \"kubernetes.io/projected/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-kube-api-access-ztgm9\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:18.926749 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:18.926725    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret podName:e5e01f7a-8ff8-4821-b662-aed4b0d11377 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:19.426707822 +0000 UTC m=+2.997006145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret") pod "global-pull-secret-syncer-mp9fn" (UID: "e5e01f7a-8ff8-4821-b662-aed4b0d11377") : object "kube-system"/"original-pull-secret" not registered
Sep 06 19:27:18.927079 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926750    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f-host\") pod \"node-ca-z9s22\" (UID: \"bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f\") " pod="openshift-image-registry/node-ca-z9s22"
Sep 06 19:27:18.927079 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926766    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-sysconfig\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.927079 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926782    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.927079 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926821    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.927079 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926860    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n56n\" (UniqueName: \"kubernetes.io/projected/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-kube-api-access-4n56n\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.927079 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926900    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnqgh\" (UniqueName: \"kubernetes.io/projected/28fce4a0-da8c-4895-9d5f-375eb510b966-kube-api-access-wnqgh\") pod \"iptables-alerter-dmtzl\" (UID: \"28fce4a0-da8c-4895-9d5f-375eb510b966\") " pod="openshift-network-operator/iptables-alerter-dmtzl"
Sep 06 19:27:18.927079 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926926    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/684b35b0-78f1-4eb5-9d65-2a5b5caf6464-hosts-file\") pod \"node-resolver-fs4k9\" (UID: \"684b35b0-78f1-4eb5-9d65-2a5b5caf6464\") " pod="openshift-dns/node-resolver-fs4k9"
Sep 06 19:27:18.927079 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926950    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-slash\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.927079 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.926991    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28fce4a0-da8c-4895-9d5f-375eb510b966-iptables-alerter-script\") pod \"iptables-alerter-dmtzl\" (UID: \"28fce4a0-da8c-4895-9d5f-375eb510b966\") " pod="openshift-network-operator/iptables-alerter-dmtzl"
Sep 06 19:27:18.927079 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927015    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-cni-netd\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.927079 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927036    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0132c00f-9104-4b41-a700-799dc067937d-ovn-node-metrics-cert\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.927079 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927059    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-sys-fs\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:18.927079 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927084    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-host\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927131    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-sysconfig\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927146    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-host\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927158    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs\") pod \"network-metrics-daemon-pghst\" (UID: \"abf4350f-c25e-43ae-9271-5987625590f7\") " pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927184    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/349a592c-1bd1-461a-912a-25932cf842cc-agent-certs\") pod \"konnectivity-agent-rkkmb\" (UID: \"349a592c-1bd1-461a-912a-25932cf842cc\") " pod="kube-system/konnectivity-agent-rkkmb"
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:18.927261    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:18.927317    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs podName:abf4350f-c25e-43ae-9271-5987625590f7 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:19.427299719 +0000 UTC m=+2.997598067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs") pod "network-metrics-daemon-pghst" (UID: "abf4350f-c25e-43ae-9271-5987625590f7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927317    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927341    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927363    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/349a592c-1bd1-461a-912a-25932cf842cc-konnectivity-ca\") pod \"konnectivity-agent-rkkmb\" (UID: \"349a592c-1bd1-461a-912a-25932cf842cc\") " pod="kube-system/konnectivity-agent-rkkmb"
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927393    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0132c00f-9104-4b41-a700-799dc067937d-ovnkube-script-lib\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927529    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-device-dir\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927564    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f-serviceca\") pod \"node-ca-z9s22\" (UID: \"bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f\") " pod="openshift-image-registry/node-ca-z9s22"
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927594    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-kubernetes\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927623    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927650    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-run-netns\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927670    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-var-lib-kubelet\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.927713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927696    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-cni-binary-copy\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927696    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-kubernetes\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927715    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0132c00f-9104-4b41-a700-799dc067937d-env-overrides\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927764    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frjx6\" (UniqueName: \"kubernetes.io/projected/0132c00f-9104-4b41-a700-799dc067937d-kube-api-access-frjx6\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927785    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-registration-dir\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927807    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-etc-selinux\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927843    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927827    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-var-lib-cni-multus\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927856    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/349a592c-1bd1-461a-912a-25932cf842cc-konnectivity-ca\") pod \"konnectivity-agent-rkkmb\" (UID: \"349a592c-1bd1-461a-912a-25932cf842cc\") " pod="kube-system/konnectivity-agent-rkkmb"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927894    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5hnj\" (UniqueName: \"kubernetes.io/projected/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-kube-api-access-f5hnj\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927926    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-var-lib-kubelet\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.927980    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28fce4a0-da8c-4895-9d5f-375eb510b966-host-slash\") pod \"iptables-alerter-dmtzl\" (UID: \"28fce4a0-da8c-4895-9d5f-375eb510b966\") " pod="openshift-network-operator/iptables-alerter-dmtzl"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928008    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-cni-bin\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928040    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqsxn\" (UniqueName: \"kubernetes.io/projected/bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f-kube-api-access-nqsxn\") pod \"node-ca-z9s22\" (UID: \"bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f\") " pod="openshift-image-registry/node-ca-z9s22"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928070    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-cni-binary-copy\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928092    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-cnibin\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928110    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-run-ovn\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.928502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928136    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684b35b0-78f1-4eb5-9d65-2a5b5caf6464-tmp-dir\") pod \"node-resolver-fs4k9\" (UID: \"684b35b0-78f1-4eb5-9d65-2a5b5caf6464\") " pod="openshift-dns/node-resolver-fs4k9"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928161    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-run\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928180    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-etc-kubernetes\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928218    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-var-lib-openvswitch\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928233    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-run\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928239    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-node-log\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928283    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-sysctl-conf\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928312    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-sys\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928334    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e5e01f7a-8ff8-4821-b662-aed4b0d11377-kubelet-config\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928357    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-multus-cni-dir\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928371    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-sys\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928384    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-run-netns\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928403    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-hostroot\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928424    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-multus-daemon-config\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928458    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-sysctl-conf\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928466    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-run-multus-certs\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928490    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-run-ovn-kubernetes\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.929313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928503    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e5e01f7a-8ff8-4821-b662-aed4b0d11377-kubelet-config\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:18.930224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928507    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-cni-binary-copy\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.930224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928515    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsq86\" (UniqueName: \"kubernetes.io/projected/684b35b0-78f1-4eb5-9d65-2a5b5caf6464-kube-api-access-rsq86\") pod \"node-resolver-fs4k9\" (UID: \"684b35b0-78f1-4eb5-9d65-2a5b5caf6464\") " pod="openshift-dns/node-resolver-fs4k9"
Sep 06 19:27:18.930224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928547    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdgt\" (UniqueName: \"kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt\") pod \"network-check-target-5b6qn\" (UID: \"11680e74-f55f-4749-a0a2-1180c6595524\") " pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:18.930224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928572    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-sysctl-d\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.930224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928599    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-system-cni-dir\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.930224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928626    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-run-k8s-cni-cncf-io\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:18.930224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928650    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-run-systemd\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.930224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928680    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0132c00f-9104-4b41-a700-799dc067937d-ovnkube-config\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:18.930224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928705    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-lib-modules\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.930224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928756    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-sysctl-d\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.930224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.928854    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-lib-modules\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.930932 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.930910    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-tmp\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.931125 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.931096    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-etc-tuned\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.931179 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.931134    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/349a592c-1bd1-461a-912a-25932cf842cc-agent-certs\") pod \"konnectivity-agent-rkkmb\" (UID: \"349a592c-1bd1-461a-912a-25932cf842cc\") " pod="kube-system/konnectivity-agent-rkkmb"
Sep 06 19:27:18.948264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.948236    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z6bq\" (UniqueName: \"kubernetes.io/projected/9b7ed4ce-2a6f-40c9-bcdc-70b263032d86-kube-api-access-9z6bq\") pod \"tuned-tknwp\" (UID: \"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86\") " pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:18.961318 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:18.961288    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 19:27:18.961318 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:18.961314    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 19:27:18.961318 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:18.961328    2571 projected.go:196] Error preparing data for projected volume kube-api-access-9sdgt for pod openshift-network-diagnostics/network-check-target-5b6qn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:27:18.961630 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:18.961403    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt podName:11680e74-f55f-4749-a0a2-1180c6595524 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:19.461383316 +0000 UTC m=+3.031681656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9sdgt" (UniqueName: "kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt") pod "network-check-target-5b6qn" (UID: "11680e74-f55f-4749-a0a2-1180c6595524") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:27:18.962609 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.962581    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n56n\" (UniqueName: \"kubernetes.io/projected/e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94-kube-api-access-4n56n\") pod \"multus-additional-cni-plugins-xp729\" (UID: \"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94\") " pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:18.963484 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:18.963465    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z64dh\" (UniqueName: \"kubernetes.io/projected/abf4350f-c25e-43ae-9271-5987625590f7-kube-api-access-z64dh\") pod \"network-metrics-daemon-pghst\" (UID: \"abf4350f-c25e-43ae-9271-5987625590f7\") " pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:19.029417 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029382    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-system-cni-dir\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.029417 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029414    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-run-k8s-cni-cncf-io\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.029647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029430    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-run-systemd\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.029647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029445    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0132c00f-9104-4b41-a700-799dc067937d-ovnkube-config\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.029647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029464    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-multus-conf-dir\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.029647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029488    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-run-openvswitch\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.029647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029513    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-run-systemd\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.029647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029515    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-log-socket\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.029647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029555    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-log-socket\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.029647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029513    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-run-k8s-cni-cncf-io\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.029647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029566    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.029647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029555    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-multus-conf-dir\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.029647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029513    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-system-cni-dir\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.029647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029599    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-multus-socket-dir-parent\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.029647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029643    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-multus-socket-dir-parent\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.029647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029647    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-kubelet\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.029647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029637    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029673    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-systemd-units\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029680    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-kubelet\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029698    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-etc-openvswitch\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029713    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-systemd-units\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029725    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-var-lib-cni-bin\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029757    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-var-lib-cni-bin\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029730    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-etc-openvswitch\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029759    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-socket-dir\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029725    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-run-openvswitch\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029801    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-var-lib-kubelet\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029830    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-var-lib-kubelet\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029851    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-socket-dir\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029855    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029896    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztgm9\" (UniqueName: \"kubernetes.io/projected/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-kube-api-access-ztgm9\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029919    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f-host\") pod \"node-ca-z9s22\" (UID: \"bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f\") " pod="openshift-image-registry/node-ca-z9s22"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029943    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnqgh\" (UniqueName: \"kubernetes.io/projected/28fce4a0-da8c-4895-9d5f-375eb510b966-kube-api-access-wnqgh\") pod \"iptables-alerter-dmtzl\" (UID: \"28fce4a0-da8c-4895-9d5f-375eb510b966\") " pod="openshift-network-operator/iptables-alerter-dmtzl"
Sep 06 19:27:19.030395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029971    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/684b35b0-78f1-4eb5-9d65-2a5b5caf6464-hosts-file\") pod \"node-resolver-fs4k9\" (UID: \"684b35b0-78f1-4eb5-9d65-2a5b5caf6464\") " pod="openshift-dns/node-resolver-fs4k9"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029986    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-slash\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030000    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28fce4a0-da8c-4895-9d5f-375eb510b966-iptables-alerter-script\") pod \"iptables-alerter-dmtzl\" (UID: \"28fce4a0-da8c-4895-9d5f-375eb510b966\") " pod="openshift-network-operator/iptables-alerter-dmtzl"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030015    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-cni-netd\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030029    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0132c00f-9104-4b41-a700-799dc067937d-ovn-node-metrics-cert\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030051    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-sys-fs\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030077    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0132c00f-9104-4b41-a700-799dc067937d-ovnkube-script-lib\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030101    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-device-dir\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030122    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0132c00f-9104-4b41-a700-799dc067937d-ovnkube-config\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030126    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f-serviceca\") pod \"node-ca-z9s22\" (UID: \"bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f\") " pod="openshift-image-registry/node-ca-z9s22"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030171    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f-host\") pod \"node-ca-z9s22\" (UID: \"bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f\") " pod="openshift-image-registry/node-ca-z9s22"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030184    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-cni-netd\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030174    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-run-netns\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030209    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-run-netns\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030241    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-cni-binary-copy\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030247    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-device-dir\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030266    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0132c00f-9104-4b41-a700-799dc067937d-env-overrides\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.031245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030290    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frjx6\" (UniqueName: \"kubernetes.io/projected/0132c00f-9104-4b41-a700-799dc067937d-kube-api-access-frjx6\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030315    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-registration-dir\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030344    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-etc-selinux\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030389    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-var-lib-cni-multus\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030413    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5hnj\" (UniqueName: \"kubernetes.io/projected/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-kube-api-access-f5hnj\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030438    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28fce4a0-da8c-4895-9d5f-375eb510b966-host-slash\") pod \"iptables-alerter-dmtzl\" (UID: \"28fce4a0-da8c-4895-9d5f-375eb510b966\") " pod="openshift-network-operator/iptables-alerter-dmtzl"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030461    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-cni-bin\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030484    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqsxn\" (UniqueName: \"kubernetes.io/projected/bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f-kube-api-access-nqsxn\") pod \"node-ca-z9s22\" (UID: \"bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f\") " pod="openshift-image-registry/node-ca-z9s22"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030515    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-cnibin\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030537    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-run-ovn\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030590    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684b35b0-78f1-4eb5-9d65-2a5b5caf6464-tmp-dir\") pod \"node-resolver-fs4k9\" (UID: \"684b35b0-78f1-4eb5-9d65-2a5b5caf6464\") " pod="openshift-dns/node-resolver-fs4k9"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030602    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-sys-fs\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030618    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-etc-kubernetes\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.029998    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030643    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-var-lib-openvswitch\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030665    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-node-log\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030682    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/684b35b0-78f1-4eb5-9d65-2a5b5caf6464-hosts-file\") pod \"node-resolver-fs4k9\" (UID: \"684b35b0-78f1-4eb5-9d65-2a5b5caf6464\") " pod="openshift-dns/node-resolver-fs4k9"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030687    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0132c00f-9104-4b41-a700-799dc067937d-ovnkube-script-lib\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.031983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030691    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-multus-cni-dir\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030718    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-etc-selinux\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030721    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-slash\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030743    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-run-netns\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030808    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-hostroot\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030822    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-multus-cni-dir\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030836    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-multus-daemon-config\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030863    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-run-multus-certs\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030867    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-var-lib-cni-multus\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030888    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-run-ovn-kubernetes\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030916    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsq86\" (UniqueName: \"kubernetes.io/projected/684b35b0-78f1-4eb5-9d65-2a5b5caf6464-kube-api-access-rsq86\") pod \"node-resolver-fs4k9\" (UID: \"684b35b0-78f1-4eb5-9d65-2a5b5caf6464\") " pod="openshift-dns/node-resolver-fs4k9"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030947    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-registration-dir\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031023    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-run-netns\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031065    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-cni-bin\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031098    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-cnibin\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031106    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-hostroot\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031177    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28fce4a0-da8c-4895-9d5f-375eb510b966-iptables-alerter-script\") pod \"iptables-alerter-dmtzl\" (UID: \"28fce4a0-da8c-4895-9d5f-375eb510b966\") " pod="openshift-network-operator/iptables-alerter-dmtzl"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031320    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684b35b0-78f1-4eb5-9d65-2a5b5caf6464-tmp-dir\") pod \"node-resolver-fs4k9\" (UID: \"684b35b0-78f1-4eb5-9d65-2a5b5caf6464\") " pod="openshift-dns/node-resolver-fs4k9"
Sep 06 19:27:19.032641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031521    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0132c00f-9104-4b41-a700-799dc067937d-env-overrides\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.033285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031529    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-etc-kubernetes\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.033285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.030621    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f-serviceca\") pod \"node-ca-z9s22\" (UID: \"bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f\") " pod="openshift-image-registry/node-ca-z9s22"
Sep 06 19:27:19.033285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031593    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-host-run-ovn-kubernetes\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.033285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031604    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-cni-binary-copy\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.033285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031595    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-var-lib-openvswitch\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.033285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031629    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-node-log\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.033285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031634    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-host-run-multus-certs\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.033285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031643    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-multus-daemon-config\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.033285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031024    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28fce4a0-da8c-4895-9d5f-375eb510b966-host-slash\") pod \"iptables-alerter-dmtzl\" (UID: \"28fce4a0-da8c-4895-9d5f-375eb510b966\") " pod="openshift-network-operator/iptables-alerter-dmtzl"
Sep 06 19:27:19.033285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.031666    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0132c00f-9104-4b41-a700-799dc067937d-run-ovn\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.033847 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.033822    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0132c00f-9104-4b41-a700-799dc067937d-ovn-node-metrics-cert\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.044197 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.044127    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztgm9\" (UniqueName: \"kubernetes.io/projected/fafb9ce0-7b24-4e74-b8ab-4d11cec62a07-kube-api-access-ztgm9\") pod \"aws-ebs-csi-driver-node-nv7jv\" (UID: \"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:19.044322 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.044306    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsq86\" (UniqueName: \"kubernetes.io/projected/684b35b0-78f1-4eb5-9d65-2a5b5caf6464-kube-api-access-rsq86\") pod \"node-resolver-fs4k9\" (UID: \"684b35b0-78f1-4eb5-9d65-2a5b5caf6464\") " pod="openshift-dns/node-resolver-fs4k9"
Sep 06 19:27:19.044412 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.044393    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqsxn\" (UniqueName: \"kubernetes.io/projected/bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f-kube-api-access-nqsxn\") pod \"node-ca-z9s22\" (UID: \"bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f\") " pod="openshift-image-registry/node-ca-z9s22"
Sep 06 19:27:19.044483 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.044405    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnqgh\" (UniqueName: \"kubernetes.io/projected/28fce4a0-da8c-4895-9d5f-375eb510b966-kube-api-access-wnqgh\") pod \"iptables-alerter-dmtzl\" (UID: \"28fce4a0-da8c-4895-9d5f-375eb510b966\") " pod="openshift-network-operator/iptables-alerter-dmtzl"
Sep 06 19:27:19.045746 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.045715    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frjx6\" (UniqueName: \"kubernetes.io/projected/0132c00f-9104-4b41-a700-799dc067937d-kube-api-access-frjx6\") pod \"ovnkube-node-n58nm\" (UID: \"0132c00f-9104-4b41-a700-799dc067937d\") " pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.046109 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.046092    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5hnj\" (UniqueName: \"kubernetes.io/projected/00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a-kube-api-access-f5hnj\") pod \"multus-qrvg8\" (UID: \"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a\") " pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.111810 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.111774    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xp729"
Sep 06 19:27:19.121097 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.121072    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rkkmb"
Sep 06 19:27:19.130966 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.130941    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tknwp"
Sep 06 19:27:19.137738 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.137716    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dmtzl"
Sep 06 19:27:19.142952 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.142932    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qrvg8"
Sep 06 19:27:19.148697 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.148675    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:19.154459 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.154429    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv"
Sep 06 19:27:19.161032 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.161012    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fs4k9"
Sep 06 19:27:19.166643 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.166622    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z9s22"
Sep 06 19:27:19.208656 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.208628    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 19:27:19.434378 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.434289    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:19.434378 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.434331    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs\") pod \"network-metrics-daemon-pghst\" (UID: \"abf4350f-c25e-43ae-9271-5987625590f7\") " pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:19.434607 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:19.434454    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:27:19.434607 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:19.434527    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs podName:abf4350f-c25e-43ae-9271-5987625590f7 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:20.434509838 +0000 UTC m=+4.004808165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs") pod "network-metrics-daemon-pghst" (UID: "abf4350f-c25e-43ae-9271-5987625590f7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:27:19.434607 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:19.434455    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 19:27:19.434607 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:19.434605    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret podName:e5e01f7a-8ff8-4821-b662-aed4b0d11377 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:20.434588162 +0000 UTC m=+4.004886499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret") pod "global-pull-secret-syncer-mp9fn" (UID: "e5e01f7a-8ff8-4821-b662-aed4b0d11377") : object "kube-system"/"original-pull-secret" not registered
Sep 06 19:27:19.535245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.535206    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdgt\" (UniqueName: \"kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt\") pod \"network-check-target-5b6qn\" (UID: \"11680e74-f55f-4749-a0a2-1180c6595524\") " pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:19.535435 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:19.535364    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 19:27:19.535435 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:19.535386    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 19:27:19.535435 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:19.535396    2571 projected.go:196] Error preparing data for projected volume kube-api-access-9sdgt for pod openshift-network-diagnostics/network-check-target-5b6qn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:27:19.535596 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:19.535450    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt podName:11680e74-f55f-4749-a0a2-1180c6595524 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:20.535434288 +0000 UTC m=+4.105732613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9sdgt" (UniqueName: "kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt") pod "network-check-target-5b6qn" (UID: "11680e74-f55f-4749-a0a2-1180c6595524") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:27:19.607673 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.607638    2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 19:27:19.859005 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.858890    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 19:22:18 +0000 UTC" deadline="2028-03-11 22:47:46.864468953 +0000 UTC"
Sep 06 19:27:19.859005 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.858938    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13251h20m27.005533637s"
Sep 06 19:27:19.877044 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:19.877009    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf11e6d5_1dfb_4b4c_b01a_f5ad0c6fc91f.slice/crio-06074dbcbe130e754a14058f391d124698f98b3e44d557766b01f241919b3241 WatchSource:0}: Error finding container 06074dbcbe130e754a14058f391d124698f98b3e44d557766b01f241919b3241: Status 404 returned error can't find the container with id 06074dbcbe130e754a14058f391d124698f98b3e44d557766b01f241919b3241
Sep 06 19:27:19.878558 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:19.878459    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafb9ce0_7b24_4e74_b8ab_4d11cec62a07.slice/crio-d10b61cf822826e576de1b52e043e23973b2d914d57e943eb6f3ffbc84d0ddff WatchSource:0}: Error finding container d10b61cf822826e576de1b52e043e23973b2d914d57e943eb6f3ffbc84d0ddff: Status 404 returned error can't find the container with id d10b61cf822826e576de1b52e043e23973b2d914d57e943eb6f3ffbc84d0ddff
Sep 06 19:27:19.879034 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:19.879003    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e9861f_8b5d_4ed4_b1be_777aa3ae2d0a.slice/crio-151ed8205d2f5f4eb7bc17775adccad4e86b7e2d43055b8bf53377ab12dc28ca WatchSource:0}: Error finding container 151ed8205d2f5f4eb7bc17775adccad4e86b7e2d43055b8bf53377ab12dc28ca: Status 404 returned error can't find the container with id 151ed8205d2f5f4eb7bc17775adccad4e86b7e2d43055b8bf53377ab12dc28ca
Sep 06 19:27:19.879996 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:19.879871    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0132c00f_9104_4b41_a700_799dc067937d.slice/crio-f9ed3d01e1a414ab9b0098a7763fd01e9b0525ac4247c6f221692a3cd298a3f3 WatchSource:0}: Error finding container f9ed3d01e1a414ab9b0098a7763fd01e9b0525ac4247c6f221692a3cd298a3f3: Status 404 returned error can't find the container with id f9ed3d01e1a414ab9b0098a7763fd01e9b0525ac4247c6f221692a3cd298a3f3
Sep 06 19:27:19.884993 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:19.884939    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684b35b0_78f1_4eb5_9d65_2a5b5caf6464.slice/crio-ac47c6d07f93a4221df129356dc64f847d8b8d8fbfc119d5ba0a259a5150c9ba WatchSource:0}: Error finding container ac47c6d07f93a4221df129356dc64f847d8b8d8fbfc119d5ba0a259a5150c9ba: Status 404 returned error can't find the container with id ac47c6d07f93a4221df129356dc64f847d8b8d8fbfc119d5ba0a259a5150c9ba
Sep 06 19:27:19.886125 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:19.886099    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e8b6e7_c76c_4775_8a5c_4594f0a1ca94.slice/crio-66371feebebbb46e39b564d768972577e6b1d50be6009e2e8b1587ce85b62307 WatchSource:0}: Error finding container 66371feebebbb46e39b564d768972577e6b1d50be6009e2e8b1587ce85b62307: Status 404 returned error can't find the container with id 66371feebebbb46e39b564d768972577e6b1d50be6009e2e8b1587ce85b62307
Sep 06 19:27:19.889120 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:19.889080    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7ed4ce_2a6f_40c9_bcdc_70b263032d86.slice/crio-cd9dda779a6c316a660c2a7624c6162415d6fdc9d7b5a18a7e39111128637b0d WatchSource:0}: Error finding container cd9dda779a6c316a660c2a7624c6162415d6fdc9d7b5a18a7e39111128637b0d: Status 404 returned error can't find the container with id cd9dda779a6c316a660c2a7624c6162415d6fdc9d7b5a18a7e39111128637b0d
Sep 06 19:27:19.890289 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:19.890266    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fce4a0_da8c_4895_9d5f_375eb510b966.slice/crio-24f91227fa20e02fb3933144f898f46c48eb41aa3de13b5cd4825adf6a9cd509 WatchSource:0}: Error finding container 24f91227fa20e02fb3933144f898f46c48eb41aa3de13b5cd4825adf6a9cd509: Status 404 returned error can't find the container with id 24f91227fa20e02fb3933144f898f46c48eb41aa3de13b5cd4825adf6a9cd509
Sep 06 19:27:19.925274 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.925237    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tknwp" event={"ID":"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86","Type":"ContainerStarted","Data":"cd9dda779a6c316a660c2a7624c6162415d6fdc9d7b5a18a7e39111128637b0d"}
Sep 06 19:27:19.926485 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.926460    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm" event={"ID":"0132c00f-9104-4b41-a700-799dc067937d","Type":"ContainerStarted","Data":"f9ed3d01e1a414ab9b0098a7763fd01e9b0525ac4247c6f221692a3cd298a3f3"}
Sep 06 19:27:19.927591 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.927558    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dmtzl" event={"ID":"28fce4a0-da8c-4895-9d5f-375eb510b966","Type":"ContainerStarted","Data":"24f91227fa20e02fb3933144f898f46c48eb41aa3de13b5cd4825adf6a9cd509"}
Sep 06 19:27:19.928799 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.928768    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp729" event={"ID":"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94","Type":"ContainerStarted","Data":"66371feebebbb46e39b564d768972577e6b1d50be6009e2e8b1587ce85b62307"}
Sep 06 19:27:19.929920 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.929897    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fs4k9" event={"ID":"684b35b0-78f1-4eb5-9d65-2a5b5caf6464","Type":"ContainerStarted","Data":"ac47c6d07f93a4221df129356dc64f847d8b8d8fbfc119d5ba0a259a5150c9ba"}
Sep 06 19:27:19.930829 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.930809    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qrvg8" event={"ID":"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a","Type":"ContainerStarted","Data":"151ed8205d2f5f4eb7bc17775adccad4e86b7e2d43055b8bf53377ab12dc28ca"}
Sep 06 19:27:19.934301 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.934274    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rkkmb" event={"ID":"349a592c-1bd1-461a-912a-25932cf842cc","Type":"ContainerStarted","Data":"2aff81cba4ab6512a6f92c8b3417e425a00c8dce4b47563fdec514e965eccbd8"}
Sep 06 19:27:19.938358 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.938333    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv" event={"ID":"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07","Type":"ContainerStarted","Data":"d10b61cf822826e576de1b52e043e23973b2d914d57e943eb6f3ffbc84d0ddff"}
Sep 06 19:27:19.939427 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:19.939406    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z9s22" event={"ID":"bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f","Type":"ContainerStarted","Data":"06074dbcbe130e754a14058f391d124698f98b3e44d557766b01f241919b3241"}
Sep 06 19:27:20.442343 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:20.442229    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:20.442343 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:20.442325    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs\") pod \"network-metrics-daemon-pghst\" (UID: \"abf4350f-c25e-43ae-9271-5987625590f7\") " pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:20.442567 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:20.442450    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:27:20.442567 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:20.442519    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs podName:abf4350f-c25e-43ae-9271-5987625590f7 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:22.442500644 +0000 UTC m=+6.012798986 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs") pod "network-metrics-daemon-pghst" (UID: "abf4350f-c25e-43ae-9271-5987625590f7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:27:20.442684 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:20.442581    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 19:27:20.442684 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:20.442619    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret podName:e5e01f7a-8ff8-4821-b662-aed4b0d11377 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:22.442605599 +0000 UTC m=+6.012903923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret") pod "global-pull-secret-syncer-mp9fn" (UID: "e5e01f7a-8ff8-4821-b662-aed4b0d11377") : object "kube-system"/"original-pull-secret" not registered
Sep 06 19:27:20.543865 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:20.543215    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdgt\" (UniqueName: \"kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt\") pod \"network-check-target-5b6qn\" (UID: \"11680e74-f55f-4749-a0a2-1180c6595524\") " pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:20.543865 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:20.543406    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 19:27:20.543865 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:20.543427    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 19:27:20.543865 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:20.543442    2571 projected.go:196] Error preparing data for projected volume kube-api-access-9sdgt for pod openshift-network-diagnostics/network-check-target-5b6qn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:27:20.543865 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:20.543504    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt podName:11680e74-f55f-4749-a0a2-1180c6595524 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:22.543486238 +0000 UTC m=+6.113784577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9sdgt" (UniqueName: "kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt") pod "network-check-target-5b6qn" (UID: "11680e74-f55f-4749-a0a2-1180c6595524") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:27:20.859363 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:20.859260    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 19:22:18 +0000 UTC" deadline="2028-06-23 14:17:10.528959266 +0000 UTC"
Sep 06 19:27:20.859363 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:20.859304    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15738h49m49.669659736s"
Sep 06 19:27:20.925991 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:20.925210    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:20.925991 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:20.925346    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:20.925991 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:20.925763    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:20.925991 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:20.925851    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:20.928800 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:20.926462    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:20.928800 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:20.926570    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:20.961675 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:20.960913    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-42.ec2.internal" event={"ID":"46e2ca63eb83f4b4b12734788288736a","Type":"ContainerStarted","Data":"2fd7c5360e717a7f3de0d1b7259a16769bb05224693aa6064537c93b3ecea75c"}
Sep 06 19:27:20.975103 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:20.973443    2571 generic.go:356] "Generic (PLEG): container finished" podID="bee41a806707e9c28c0a2c777078070a" containerID="2759de91a9bd9643e4bc5069d3ec5de64359c2dcc5ab99b75cb0d61de3d66509" exitCode=0
Sep 06 19:27:20.976484 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:20.976453    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal" event={"ID":"bee41a806707e9c28c0a2c777078070a","Type":"ContainerDied","Data":"2759de91a9bd9643e4bc5069d3ec5de64359c2dcc5ab99b75cb0d61de3d66509"}
Sep 06 19:27:20.991025 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:20.990939    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-42.ec2.internal" podStartSLOduration=2.990918931 podStartE2EDuration="2.990918931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:20.973148903 +0000 UTC m=+4.543447250" watchObservedRunningTime="2026-09-06 19:27:20.990918931 +0000 UTC m=+4.561217280"
Sep 06 19:27:22.005758 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:22.005502    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal" event={"ID":"bee41a806707e9c28c0a2c777078070a","Type":"ContainerStarted","Data":"3c6704726793b29ccc23afec7bf023a230477a5e7809823e15d5e7d047aa93b2"}
Sep 06 19:27:22.020172 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:22.019468    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-42.ec2.internal" podStartSLOduration=4.019445041 podStartE2EDuration="4.019445041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:22.019178845 +0000 UTC m=+5.589477192" watchObservedRunningTime="2026-09-06 19:27:22.019445041 +0000 UTC m=+5.589743388"
Sep 06 19:27:22.464684 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:22.463778    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:22.464684 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:22.463843    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs\") pod \"network-metrics-daemon-pghst\" (UID: \"abf4350f-c25e-43ae-9271-5987625590f7\") " pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:22.464684 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:22.464019    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:27:22.464684 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:22.464078    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs podName:abf4350f-c25e-43ae-9271-5987625590f7 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:26.464060817 +0000 UTC m=+10.034359156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs") pod "network-metrics-daemon-pghst" (UID: "abf4350f-c25e-43ae-9271-5987625590f7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:27:22.464684 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:22.464472    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 19:27:22.464684 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:22.464521    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret podName:e5e01f7a-8ff8-4821-b662-aed4b0d11377 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:26.464506586 +0000 UTC m=+10.034804924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret") pod "global-pull-secret-syncer-mp9fn" (UID: "e5e01f7a-8ff8-4821-b662-aed4b0d11377") : object "kube-system"/"original-pull-secret" not registered
Sep 06 19:27:22.565795 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:22.565185    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdgt\" (UniqueName: \"kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt\") pod \"network-check-target-5b6qn\" (UID: \"11680e74-f55f-4749-a0a2-1180c6595524\") " pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:22.565795 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:22.565354    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 19:27:22.565795 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:22.565379    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 19:27:22.565795 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:22.565395    2571 projected.go:196] Error preparing data for projected volume kube-api-access-9sdgt for pod openshift-network-diagnostics/network-check-target-5b6qn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:27:22.565795 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:22.565458    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt podName:11680e74-f55f-4749-a0a2-1180c6595524 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:26.565437705 +0000 UTC m=+10.135736044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9sdgt" (UniqueName: "kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt") pod "network-check-target-5b6qn" (UID: "11680e74-f55f-4749-a0a2-1180c6595524") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:27:22.914715 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:22.914630    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:22.914869 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:22.914758    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:22.915159 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:22.915141    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:22.915253 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:22.915234    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:22.915316 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:22.915303    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:22.915412 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:22.915393    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:24.914700 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:24.914664    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:24.915196 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:24.914825    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:24.915264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:24.915218    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:24.915331 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:24.915311    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:24.915386 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:24.915364    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:24.915456 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:24.915438    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:26.501638 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:26.501598    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:26.502146 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:26.501658    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs\") pod \"network-metrics-daemon-pghst\" (UID: \"abf4350f-c25e-43ae-9271-5987625590f7\") " pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:26.502146 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:26.501771    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 19:27:26.502146 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:26.501837    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret podName:e5e01f7a-8ff8-4821-b662-aed4b0d11377 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:34.501817101 +0000 UTC m=+18.072115427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret") pod "global-pull-secret-syncer-mp9fn" (UID: "e5e01f7a-8ff8-4821-b662-aed4b0d11377") : object "kube-system"/"original-pull-secret" not registered
Sep 06 19:27:26.502146 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:26.501771    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:27:26.502146 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:26.501904    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs podName:abf4350f-c25e-43ae-9271-5987625590f7 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:34.501890917 +0000 UTC m=+18.072189254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs") pod "network-metrics-daemon-pghst" (UID: "abf4350f-c25e-43ae-9271-5987625590f7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:27:26.603135 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:26.603006    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdgt\" (UniqueName: \"kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt\") pod \"network-check-target-5b6qn\" (UID: \"11680e74-f55f-4749-a0a2-1180c6595524\") " pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:26.603316 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:26.603195    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 19:27:26.603316 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:26.603224    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 19:27:26.603316 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:26.603239    2571 projected.go:196] Error preparing data for projected volume kube-api-access-9sdgt for pod openshift-network-diagnostics/network-check-target-5b6qn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:27:26.603316 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:26.603303    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt podName:11680e74-f55f-4749-a0a2-1180c6595524 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:34.603280882 +0000 UTC m=+18.173579215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9sdgt" (UniqueName: "kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt") pod "network-check-target-5b6qn" (UID: "11680e74-f55f-4749-a0a2-1180c6595524") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:27:26.913538 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:26.913460    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:26.913692 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:26.913577    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:26.914218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:26.913873    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:26.914218 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:26.914021    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:26.914218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:26.914088    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:26.914218 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:26.914187    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:28.912796 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:28.912749    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:28.913275 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:28.912749    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:28.913275 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:28.912872    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:28.913275 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:28.913039    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:28.913275 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:28.912771    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:28.913275 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:28.913137    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:30.912217 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:30.912076    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:30.912646 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:30.912076    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:30.912646 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:30.912223    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:30.912646 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:30.912251    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:30.912646 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:30.912345    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:30.912646 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:30.912432    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:32.912105 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:32.912075    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:32.912105 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:32.912092    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:32.912643 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:32.912231    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:32.912643 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:32.912288    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:32.912643 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:32.912409    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:32.912643 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:32.912507    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:34.563622 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:34.563573    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:34.564086 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:34.563641    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs\") pod \"network-metrics-daemon-pghst\" (UID: \"abf4350f-c25e-43ae-9271-5987625590f7\") " pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:34.564086 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:34.563738    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 19:27:34.564086 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:34.563814    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret podName:e5e01f7a-8ff8-4821-b662-aed4b0d11377 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:50.563793287 +0000 UTC m=+34.134091659 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret") pod "global-pull-secret-syncer-mp9fn" (UID: "e5e01f7a-8ff8-4821-b662-aed4b0d11377") : object "kube-system"/"original-pull-secret" not registered
Sep 06 19:27:34.564086 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:34.563820    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:27:34.564086 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:34.563886    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs podName:abf4350f-c25e-43ae-9271-5987625590f7 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:50.563867444 +0000 UTC m=+34.134165785 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs") pod "network-metrics-daemon-pghst" (UID: "abf4350f-c25e-43ae-9271-5987625590f7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:27:34.664744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:34.664704    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdgt\" (UniqueName: \"kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt\") pod \"network-check-target-5b6qn\" (UID: \"11680e74-f55f-4749-a0a2-1180c6595524\") " pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:34.664909 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:34.664864    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 19:27:34.664909 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:34.664882    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 19:27:34.664909 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:34.664895    2571 projected.go:196] Error preparing data for projected volume kube-api-access-9sdgt for pod openshift-network-diagnostics/network-check-target-5b6qn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:27:34.665043 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:34.664975    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt podName:11680e74-f55f-4749-a0a2-1180c6595524 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:50.66494186 +0000 UTC m=+34.235240189 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9sdgt" (UniqueName: "kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt") pod "network-check-target-5b6qn" (UID: "11680e74-f55f-4749-a0a2-1180c6595524") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:27:34.912783 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:34.912682    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:34.912783 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:34.912701    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:34.912783 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:34.912681    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:34.913063 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:34.912823    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:34.913063 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:34.912903    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:34.913063 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:34.913003    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:36.912701 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:36.912664    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:36.913195 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:36.912780    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:36.913195 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:36.912787    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:36.913195 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:36.912802    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:36.913195 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:36.912900    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:36.913195 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:36.913004    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:38.041195 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.041020    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp729" event={"ID":"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94","Type":"ContainerStarted","Data":"e5264114a60eaf6a8373d64ffbbd9ef84fda072eeb03bb00161feae9dd7d4cad"}
Sep 06 19:27:38.042683 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.042659    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fs4k9" event={"ID":"684b35b0-78f1-4eb5-9d65-2a5b5caf6464","Type":"ContainerStarted","Data":"75cb15f40bb6c2524a0b4bbf320e6e23421b3c8ab484b56d6c028edc8410bdd1"}
Sep 06 19:27:38.044238 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.044171    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rkkmb" event={"ID":"349a592c-1bd1-461a-912a-25932cf842cc","Type":"ContainerStarted","Data":"8ce55e9b47bc7e3ac6aa5214c91cc9f8a7125f6db04cf55dd8e309c4f09fbef3"}
Sep 06 19:27:38.048156 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.048129    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv" event={"ID":"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07","Type":"ContainerStarted","Data":"e81c7f412652a6a7d8bc42e673e12e20f95f68244254578f9931bdad83a24004"}
Sep 06 19:27:38.050641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.050412    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z9s22" event={"ID":"bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f","Type":"ContainerStarted","Data":"ff17913aaa73d127d5410d8eba92560b190cf73ced9560a9d500e70c65d73575"}
Sep 06 19:27:38.052266 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.052183    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tknwp" event={"ID":"9b7ed4ce-2a6f-40c9-bcdc-70b263032d86","Type":"ContainerStarted","Data":"57a90c61214fc91b309f0b2576d9466f519840ff05adef8e8dc3e936d3c73469"}
Sep 06 19:27:38.054224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.054167    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm" event={"ID":"0132c00f-9104-4b41-a700-799dc067937d","Type":"ContainerStarted","Data":"dc3e2e7782b1d613c8af3959a71f49fda724478a3af32b8ac427afd65176c64f"}
Sep 06 19:27:38.072651 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.072610    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tknwp" podStartSLOduration=3.176282994 podStartE2EDuration="21.072595104s" totalImagesPullingTime="17.89631211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:38.072423013 +0000 UTC m=+21.642721359" watchObservedRunningTime="2026-09-06 19:27:38.072595104 +0000 UTC m=+21.642893506"
Sep 06 19:27:38.103329 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.103275    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fs4k9" podStartSLOduration=3.203624296 podStartE2EDuration="21.103253526s" totalImagesPullingTime="17.89962923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:38.084824104 +0000 UTC m=+21.655122451" watchObservedRunningTime="2026-09-06 19:27:38.103253526 +0000 UTC m=+21.673551872"
Sep 06 19:27:38.103426 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.103389    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z9s22" podStartSLOduration=3.477149088 podStartE2EDuration="21.103379045s" totalImagesPullingTime="17.626229957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:38.102676988 +0000 UTC m=+21.672975335" watchObservedRunningTime="2026-09-06 19:27:38.103379045 +0000 UTC m=+21.673677392"
Sep 06 19:27:38.674222 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.674179    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rkkmb"
Sep 06 19:27:38.674936 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.674912    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rkkmb"
Sep 06 19:27:38.689545 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.689495    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rkkmb" podStartSLOduration=3.806081058 podStartE2EDuration="21.689479341s" totalImagesPullingTime="17.883398283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:38.121217988 +0000 UTC m=+21.691516333" watchObservedRunningTime="2026-09-06 19:27:38.689479341 +0000 UTC m=+22.259777686"
Sep 06 19:27:38.912448 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.912417    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:38.912627 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.912417    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:38.912627 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:38.912525    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:38.912627 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:38.912537    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:38.912748 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:38.912634    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:38.912748 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:38.912728    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:39.057952 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:39.057918    2571 generic.go:356] "Generic (PLEG): container finished" podID="e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94" containerID="e5264114a60eaf6a8373d64ffbbd9ef84fda072eeb03bb00161feae9dd7d4cad" exitCode=0
Sep 06 19:27:39.058866 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:39.058011    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp729" event={"ID":"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94","Type":"ContainerDied","Data":"e5264114a60eaf6a8373d64ffbbd9ef84fda072eeb03bb00161feae9dd7d4cad"}
Sep 06 19:27:39.059667 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:39.059644    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qrvg8" event={"ID":"00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a","Type":"ContainerStarted","Data":"6d82755fa3791e13fe15780f51207789f7abdf68360c905f679b28b20dc36658"}
Sep 06 19:27:39.062638 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:39.062612    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm" event={"ID":"0132c00f-9104-4b41-a700-799dc067937d","Type":"ContainerStarted","Data":"81d72ffa96b2e2b31aae136d562fd93975fc62f6016b7d5bb6245e680fa06a27"}
Sep 06 19:27:39.062740 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:39.062679    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm" event={"ID":"0132c00f-9104-4b41-a700-799dc067937d","Type":"ContainerStarted","Data":"7aa84663c44b5ff48bbca4a371c5ce6a71c20bb84717e098b1a969a982acce20"}
Sep 06 19:27:39.062740 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:39.062708    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm" event={"ID":"0132c00f-9104-4b41-a700-799dc067937d","Type":"ContainerStarted","Data":"0b3099a138a796bf93bcadc0a882051d76e1f1e3ded4f327e1fed787699aa914"}
Sep 06 19:27:39.062740 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:39.062736    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm" event={"ID":"0132c00f-9104-4b41-a700-799dc067937d","Type":"ContainerStarted","Data":"3061898aee82af7cbb0def95fad5771f1c2b7735604ef32a7c80f27e673a2e96"}
Sep 06 19:27:39.062901 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:39.062766    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm" event={"ID":"0132c00f-9104-4b41-a700-799dc067937d","Type":"ContainerStarted","Data":"6cad3de2047708274f6abe2810bbc21334efafbfe0f567e5851b0cbc11501d4e"}
Sep 06 19:27:39.062995 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:39.062977    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rkkmb"
Sep 06 19:27:39.063442 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:39.063427    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rkkmb"
Sep 06 19:27:39.102797 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:39.102678    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qrvg8" podStartSLOduration=3.995043836 podStartE2EDuration="22.102663438s" totalImagesPullingTime="18.107619602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:39.102643134 +0000 UTC m=+22.672941481" watchObservedRunningTime="2026-09-06 19:27:39.102663438 +0000 UTC m=+22.672961783"
Sep 06 19:27:39.280186 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:39.280159    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 06 19:27:39.891069 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:39.890929    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-06T19:27:39.280179609Z","UUID":"5c89a063-b86d-4d57-a55f-093d78d0e2eb","Handler":null,"Name":"","Endpoint":""}
Sep 06 19:27:39.892972 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:39.892937    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 06 19:27:39.893120 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:39.892993    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 06 19:27:40.066936 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:40.066895    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dmtzl" event={"ID":"28fce4a0-da8c-4895-9d5f-375eb510b966","Type":"ContainerStarted","Data":"c943561df3bfafe6bb3f2d67484fbfe927f8e62d95c3d40e0f3f8726b4569cdd"}
Sep 06 19:27:40.069345 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:40.069312    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv" event={"ID":"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07","Type":"ContainerStarted","Data":"d97e934a6ee2678add22c729fe14ee7ca5416a81b4963827452bd8d2fe515ea9"}
Sep 06 19:27:40.081438 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:40.081281    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dmtzl" podStartSLOduration=5.184926294 podStartE2EDuration="23.081260794s" totalImagesPullingTime="17.8963345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:40.080695387 +0000 UTC m=+23.650993734" watchObservedRunningTime="2026-09-06 19:27:40.081260794 +0000 UTC m=+23.651559141"
Sep 06 19:27:40.912400 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:40.912308    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:40.912400 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:40.912356    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:40.912649 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:40.912308    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:40.912649 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:40.912448    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:40.912649 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:40.912524    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:40.912649 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:40.912600    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:41.074593 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:41.074556    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv" event={"ID":"fafb9ce0-7b24-4e74-b8ab-4d11cec62a07","Type":"ContainerStarted","Data":"7c62900ea2fcce1e53bd0173e31ae24d01608f3462cea7090ea416b87d112b60"}
Sep 06 19:27:41.078582 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:41.078548    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm" event={"ID":"0132c00f-9104-4b41-a700-799dc067937d","Type":"ContainerStarted","Data":"b987a51d558f131c51b05a200ccfe02ac868721833ed4fc4b4283092c8885d97"}
Sep 06 19:27:41.089405 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:41.089343    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nv7jv" podStartSLOduration=4.053328265 podStartE2EDuration="24.089322047s" totalImagesPullingTime="20.035993782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:27:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:41.088794084 +0000 UTC m=+24.659092429" watchObservedRunningTime="2026-09-06 19:27:41.089322047 +0000 UTC m=+24.659620397"
Sep 06 19:27:42.912832 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:42.912739    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:42.912832 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:42.912769    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:42.913361 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:42.912739    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:42.913361 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:42.912871    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:42.913361 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:42.912989    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:42.913361 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:42.913079    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:44.089389 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:44.089100    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm" event={"ID":"0132c00f-9104-4b41-a700-799dc067937d","Type":"ContainerStarted","Data":"0d6c9d68e0e04bdb6a3f8d2ef6cbd51574daad0654c04e52f1b0eb26cbc418b6"}
Sep 06 19:27:44.089389 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:44.089379    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:44.091170 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:44.091141    2571 generic.go:356] "Generic (PLEG): container finished" podID="e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94" containerID="43ce77459260253f8d12ff5882bc641c8a5d723a4132a8e6fae85d95968cb35d" exitCode=0
Sep 06 19:27:44.091282 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:44.091207    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp729" event={"ID":"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94","Type":"ContainerDied","Data":"43ce77459260253f8d12ff5882bc641c8a5d723a4132a8e6fae85d95968cb35d"}
Sep 06 19:27:44.104991 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:44.104951    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:44.113100 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:44.113053    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm" podStartSLOduration=9.193691263 podStartE2EDuration="27.113037613s" totalImagesPullingTime="17.91934635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:44.112503339 +0000 UTC m=+27.682801686" watchObservedRunningTime="2026-09-06 19:27:44.113037613 +0000 UTC m=+27.683335958"
Sep 06 19:27:44.912529 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:44.912497    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:44.912703 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:44.912497    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:44.912703 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:44.912621    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:44.912703 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:44.912633    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:44.912703 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:44.912679    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:44.912914 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:44.912737    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:45.096082 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:45.096054    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:45.096525 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:45.096144    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:45.120940 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:45.120916    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:27:45.319637 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:45.319608    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5b6qn"]
Sep 06 19:27:45.319845 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:45.319709    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:45.319845 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:45.319815    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:45.320247 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:45.320223    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mp9fn"]
Sep 06 19:27:45.320360 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:45.320301    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:45.320416 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:45.320395    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:45.320807 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:45.320785    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pghst"]
Sep 06 19:27:45.320871 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:45.320862    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:45.320953 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:45.320937    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:46.099193 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:46.099162    2571 generic.go:356] "Generic (PLEG): container finished" podID="e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94" containerID="a47aa2bcb56d65ee36f6b3f21dec0d43cd632d53da44a4ccd99fb75d9a442317" exitCode=0
Sep 06 19:27:46.099569 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:46.099248    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp729" event={"ID":"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94","Type":"ContainerDied","Data":"a47aa2bcb56d65ee36f6b3f21dec0d43cd632d53da44a4ccd99fb75d9a442317"}
Sep 06 19:27:46.912972 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:46.912935    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:46.913171 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:46.913039    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:46.913171 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:46.913084    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:46.913171 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:46.913077    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:46.913337 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:46.913174    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:46.913337 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:46.913235    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:47.104025 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:47.103914    2571 generic.go:356] "Generic (PLEG): container finished" podID="e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94" containerID="9bc01a6a0ce635fb00336809ee7ab7ec116438c931021a9eefb05a695d533e8a" exitCode=0
Sep 06 19:27:47.104025 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:47.104004    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp729" event={"ID":"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94","Type":"ContainerDied","Data":"9bc01a6a0ce635fb00336809ee7ab7ec116438c931021a9eefb05a695d533e8a"}
Sep 06 19:27:48.912380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:48.912345    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:48.912380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:48.912383    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:48.912993 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:48.912350    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:48.912993 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:48.912465    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp9fn" podUID="e5e01f7a-8ff8-4821-b662-aed4b0d11377"
Sep 06 19:27:48.913092 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:48.912983    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6qn" podUID="11680e74-f55f-4749-a0a2-1180c6595524"
Sep 06 19:27:48.913185 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:48.913131    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pghst" podUID="abf4350f-c25e-43ae-9271-5987625590f7"
Sep 06 19:27:50.580621 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.580406    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:50.581103 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.580644    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs\") pod \"network-metrics-daemon-pghst\" (UID: \"abf4350f-c25e-43ae-9271-5987625590f7\") " pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:50.581103 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:50.580589    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 19:27:50.581103 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:50.580770    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret podName:e5e01f7a-8ff8-4821-b662-aed4b0d11377 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:22.580753406 +0000 UTC m=+66.151051743 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret") pod "global-pull-secret-syncer-mp9fn" (UID: "e5e01f7a-8ff8-4821-b662-aed4b0d11377") : object "kube-system"/"original-pull-secret" not registered
Sep 06 19:27:50.581103 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:50.580804    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:27:50.581103 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:50.580860    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs podName:abf4350f-c25e-43ae-9271-5987625590f7 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:22.580843679 +0000 UTC m=+66.151142015 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs") pod "network-metrics-daemon-pghst" (UID: "abf4350f-c25e-43ae-9271-5987625590f7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 19:27:50.681109 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.681032    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdgt\" (UniqueName: \"kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt\") pod \"network-check-target-5b6qn\" (UID: \"11680e74-f55f-4749-a0a2-1180c6595524\") " pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:50.681267 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:50.681196    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 19:27:50.681267 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:50.681217    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 19:27:50.681267 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:50.681227    2571 projected.go:196] Error preparing data for projected volume kube-api-access-9sdgt for pod openshift-network-diagnostics/network-check-target-5b6qn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:27:50.681393 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:50.681281    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt podName:11680e74-f55f-4749-a0a2-1180c6595524 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:22.681265568 +0000 UTC m=+66.251563907 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9sdgt" (UniqueName: "kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt") pod "network-check-target-5b6qn" (UID: "11680e74-f55f-4749-a0a2-1180c6595524") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 19:27:50.700253 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.700224    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-42.ec2.internal" event="NodeReady"
Sep 06 19:27:50.700396 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.700384    2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 06 19:27:50.731185 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.731150    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"]
Sep 06 19:27:50.758570 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.758542    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nfvjj"]
Sep 06 19:27:50.758741 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.758668    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.761194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.761169    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 06 19:27:50.761528 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.761359    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 06 19:27:50.761528 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.761411    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 06 19:27:50.761764 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.761746    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sn49m\""
Sep 06 19:27:50.767925 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.767728    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 06 19:27:50.772546 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.772517    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"]
Sep 06 19:27:50.772654 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.772561    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sr895"]
Sep 06 19:27:50.772715 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.772684    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:50.774909 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.774887    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 06 19:27:50.775025 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.774887    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kglk6\""
Sep 06 19:27:50.775025 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.774936    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 06 19:27:50.794681 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.794651    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nfvjj"]
Sep 06 19:27:50.794840 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.794697    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sr895"
Sep 06 19:27:50.794840 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.794729    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sr895"]
Sep 06 19:27:50.797078 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.796879    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8xksm\""
Sep 06 19:27:50.797422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.797293    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 06 19:27:50.797422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.797316    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 06 19:27:50.797586 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.797541    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 06 19:27:50.882866 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.882833    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.883068 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.882873    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c379232e-b6a5-4ba3-b941-1232bd22c1f3-ca-trust-extracted\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.883068 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.882918    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef82340-6516-43c2-b523-c40361c19560-tmp-dir\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:50.883068 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.882980    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:50.883068 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.883015    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef82340-6516-43c2-b523-c40361c19560-config-volume\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:50.883068 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.883036    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4039f06-ffbd-4400-941c-beb1104ea6f1-cert\") pod \"ingress-canary-sr895\" (UID: \"f4039f06-ffbd-4400-941c-beb1104ea6f1\") " pod="openshift-ingress-canary/ingress-canary-sr895"
Sep 06 19:27:50.883068 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.883061    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msqgq\" (UniqueName: \"kubernetes.io/projected/f4039f06-ffbd-4400-941c-beb1104ea6f1-kube-api-access-msqgq\") pod \"ingress-canary-sr895\" (UID: \"f4039f06-ffbd-4400-941c-beb1104ea6f1\") " pod="openshift-ingress-canary/ingress-canary-sr895"
Sep 06 19:27:50.883306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.883110    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvp2m\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-kube-api-access-jvp2m\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.883306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.883135    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x72s2\" (UniqueName: \"kubernetes.io/projected/7ef82340-6516-43c2-b523-c40361c19560-kube-api-access-x72s2\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:50.883306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.883172    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-certificates\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.883306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.883215    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-bound-sa-token\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.883306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.883292    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c379232e-b6a5-4ba3-b941-1232bd22c1f3-image-registry-private-configuration\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.883489 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.883317    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c379232e-b6a5-4ba3-b941-1232bd22c1f3-trusted-ca\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.883489 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.883344    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c379232e-b6a5-4ba3-b941-1232bd22c1f3-installation-pull-secrets\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.912198 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.912161    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:27:50.912198 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.912183    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:27:50.912427 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.912162    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:27:50.915051 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.915028    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 06 19:27:50.915204 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.915028    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 06 19:27:50.915204 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.915031    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9bdz9\""
Sep 06 19:27:50.915204 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.915174    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 06 19:27:50.915390 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.915276    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 06 19:27:50.915390 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.915297    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-26tb5\""
Sep 06 19:27:50.983844 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.983804    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c379232e-b6a5-4ba3-b941-1232bd22c1f3-image-registry-private-configuration\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.984046 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.983864    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c379232e-b6a5-4ba3-b941-1232bd22c1f3-trusted-ca\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.984046 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.983892    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c379232e-b6a5-4ba3-b941-1232bd22c1f3-installation-pull-secrets\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.984046 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.983934    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.984046 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.983979    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c379232e-b6a5-4ba3-b941-1232bd22c1f3-ca-trust-extracted\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.984046 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.984019    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef82340-6516-43c2-b523-c40361c19560-tmp-dir\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:50.984304 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.984054    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:50.984304 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.984084    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef82340-6516-43c2-b523-c40361c19560-config-volume\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:50.984304 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.984105    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4039f06-ffbd-4400-941c-beb1104ea6f1-cert\") pod \"ingress-canary-sr895\" (UID: \"f4039f06-ffbd-4400-941c-beb1104ea6f1\") " pod="openshift-ingress-canary/ingress-canary-sr895"
Sep 06 19:27:50.984304 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.984127    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msqgq\" (UniqueName: \"kubernetes.io/projected/f4039f06-ffbd-4400-941c-beb1104ea6f1-kube-api-access-msqgq\") pod \"ingress-canary-sr895\" (UID: \"f4039f06-ffbd-4400-941c-beb1104ea6f1\") " pod="openshift-ingress-canary/ingress-canary-sr895"
Sep 06 19:27:50.984304 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.984158    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvp2m\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-kube-api-access-jvp2m\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.984304 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.984176    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x72s2\" (UniqueName: \"kubernetes.io/projected/7ef82340-6516-43c2-b523-c40361c19560-kube-api-access-x72s2\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:50.984304 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.984209    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-certificates\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.984304 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.984228    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-bound-sa-token\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.984804 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.984781    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c379232e-b6a5-4ba3-b941-1232bd22c1f3-trusted-ca\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.984878 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.984854    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c379232e-b6a5-4ba3-b941-1232bd22c1f3-ca-trust-extracted\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.984934 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:50.984906    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 19:27:50.984934 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:50.984922    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df7c5cf9c-nd9t8: secret "image-registry-tls" not found
Sep 06 19:27:50.985058 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:50.984983    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:27:50.985058 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:50.984994    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls podName:c379232e-b6a5-4ba3-b941-1232bd22c1f3 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:51.484976466 +0000 UTC m=+35.055274812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls") pod "image-registry-7df7c5cf9c-nd9t8" (UID: "c379232e-b6a5-4ba3-b941-1232bd22c1f3") : secret "image-registry-tls" not found
Sep 06 19:27:50.985058 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.985022    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef82340-6516-43c2-b523-c40361c19560-config-volume\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:50.985058 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:50.985036    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls podName:7ef82340-6516-43c2-b523-c40361c19560 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:51.485018903 +0000 UTC m=+35.055317244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls") pod "dns-default-nfvjj" (UID: "7ef82340-6516-43c2-b523-c40361c19560") : secret "dns-default-metrics-tls" not found
Sep 06 19:27:50.985239 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.985092    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef82340-6516-43c2-b523-c40361c19560-tmp-dir\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:50.985443 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.985416    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-certificates\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.988915 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.988891    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c379232e-b6a5-4ba3-b941-1232bd22c1f3-installation-pull-secrets\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.988915 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.988908    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c379232e-b6a5-4ba3-b941-1232bd22c1f3-image-registry-private-configuration\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.989361 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.989345    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4039f06-ffbd-4400-941c-beb1104ea6f1-cert\") pod \"ingress-canary-sr895\" (UID: \"f4039f06-ffbd-4400-941c-beb1104ea6f1\") " pod="openshift-ingress-canary/ingress-canary-sr895"
Sep 06 19:27:50.993141 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.993120    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msqgq\" (UniqueName: \"kubernetes.io/projected/f4039f06-ffbd-4400-941c-beb1104ea6f1-kube-api-access-msqgq\") pod \"ingress-canary-sr895\" (UID: \"f4039f06-ffbd-4400-941c-beb1104ea6f1\") " pod="openshift-ingress-canary/ingress-canary-sr895"
Sep 06 19:27:50.993248 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.993136    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-bound-sa-token\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:50.993248 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.993240    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x72s2\" (UniqueName: \"kubernetes.io/projected/7ef82340-6516-43c2-b523-c40361c19560-kube-api-access-x72s2\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:50.993414 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:50.993392    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvp2m\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-kube-api-access-jvp2m\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:51.114904 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:51.114874    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sr895"
Sep 06 19:27:51.430945 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:51.430918    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sr895"]
Sep 06 19:27:51.435064 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:27:51.435016    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4039f06_ffbd_4400_941c_beb1104ea6f1.slice/crio-49f84bf84e6baa0f6fe665f111f1abdf66e2434213daeaa9ec2ea0a05bc85aea WatchSource:0}: Error finding container 49f84bf84e6baa0f6fe665f111f1abdf66e2434213daeaa9ec2ea0a05bc85aea: Status 404 returned error can't find the container with id 49f84bf84e6baa0f6fe665f111f1abdf66e2434213daeaa9ec2ea0a05bc85aea
Sep 06 19:27:51.488758 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:51.488734    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:51.488875 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:51.488783    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:51.488915 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:51.488874    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:27:51.488915 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:51.488878    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 19:27:51.488915 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:51.488895    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df7c5cf9c-nd9t8: secret "image-registry-tls" not found
Sep 06 19:27:51.489047 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:51.488936    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls podName:c379232e-b6a5-4ba3-b941-1232bd22c1f3 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:52.488919294 +0000 UTC m=+36.059217634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls") pod "image-registry-7df7c5cf9c-nd9t8" (UID: "c379232e-b6a5-4ba3-b941-1232bd22c1f3") : secret "image-registry-tls" not found
Sep 06 19:27:51.489047 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:51.488950    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls podName:7ef82340-6516-43c2-b523-c40361c19560 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:52.488943162 +0000 UTC m=+36.059241486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls") pod "dns-default-nfvjj" (UID: "7ef82340-6516-43c2-b523-c40361c19560") : secret "dns-default-metrics-tls" not found
Sep 06 19:27:52.120952 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:52.120876    2571 generic.go:356] "Generic (PLEG): container finished" podID="e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94" containerID="e1feb8ee865dc902818eaf4ab7e79e4821b352612b257d0cda6dc35c7314aabe" exitCode=0
Sep 06 19:27:52.121809 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:52.120974    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp729" event={"ID":"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94","Type":"ContainerDied","Data":"e1feb8ee865dc902818eaf4ab7e79e4821b352612b257d0cda6dc35c7314aabe"}
Sep 06 19:27:52.122317 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:52.122275    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sr895" event={"ID":"f4039f06-ffbd-4400-941c-beb1104ea6f1","Type":"ContainerStarted","Data":"49f84bf84e6baa0f6fe665f111f1abdf66e2434213daeaa9ec2ea0a05bc85aea"}
Sep 06 19:27:52.498580 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:52.498546    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:52.498783 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:52.498615    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:52.498783 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:52.498707    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 19:27:52.498783 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:52.498725    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df7c5cf9c-nd9t8: secret "image-registry-tls" not found
Sep 06 19:27:52.498783 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:52.498762    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:27:52.498998 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:52.498788    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls podName:c379232e-b6a5-4ba3-b941-1232bd22c1f3 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:54.498769224 +0000 UTC m=+38.069067551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls") pod "image-registry-7df7c5cf9c-nd9t8" (UID: "c379232e-b6a5-4ba3-b941-1232bd22c1f3") : secret "image-registry-tls" not found
Sep 06 19:27:52.498998 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:52.498811    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls podName:7ef82340-6516-43c2-b523-c40361c19560 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:54.498795908 +0000 UTC m=+38.069094233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls") pod "dns-default-nfvjj" (UID: "7ef82340-6516-43c2-b523-c40361c19560") : secret "dns-default-metrics-tls" not found
Sep 06 19:27:53.129518 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:53.129477    2571 generic.go:356] "Generic (PLEG): container finished" podID="e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94" containerID="b4285d75da57b856a9e47ef9975aa106ace23e3a7f378ddd88ad20c244dd55c8" exitCode=0
Sep 06 19:27:53.129940 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:53.129547    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp729" event={"ID":"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94","Type":"ContainerDied","Data":"b4285d75da57b856a9e47ef9975aa106ace23e3a7f378ddd88ad20c244dd55c8"}
Sep 06 19:27:54.133946 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:54.133908    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sr895" event={"ID":"f4039f06-ffbd-4400-941c-beb1104ea6f1","Type":"ContainerStarted","Data":"89a8587ca6aa9ea570f51ad40b4bb91976f412c10bad6d396faa083ce460c0ae"}
Sep 06 19:27:54.137698 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:54.137671    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xp729" event={"ID":"e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94","Type":"ContainerStarted","Data":"240aaeca33a538c92e2bae658c1ba81fc3682931c82294595c2a34af77de6e32"}
Sep 06 19:27:54.148010 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:54.147945    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sr895" podStartSLOduration=1.572515459 podStartE2EDuration="4.147931899s" totalImagesPullingTime="2.57541644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:54.146771576 +0000 UTC m=+37.717069922" watchObservedRunningTime="2026-09-06 19:27:54.147931899 +0000 UTC m=+37.718230244"
Sep 06 19:27:54.165033 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:54.164983    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xp729" podStartSLOduration=9.149417951 podStartE2EDuration="37.16495035s" totalImagesPullingTime="28.015532399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:27:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:27:54.163664415 +0000 UTC m=+37.733962767" watchObservedRunningTime="2026-09-06 19:27:54.16495035 +0000 UTC m=+37.735248696"
Sep 06 19:27:54.516445 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:54.516362    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:54.516445 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:54.516415    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:54.516623 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:54.516516    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:27:54.516623 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:54.516523    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 19:27:54.516623 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:54.516541    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df7c5cf9c-nd9t8: secret "image-registry-tls" not found
Sep 06 19:27:54.516623 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:54.516582    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls podName:7ef82340-6516-43c2-b523-c40361c19560 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:58.516568717 +0000 UTC m=+42.086867041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls") pod "dns-default-nfvjj" (UID: "7ef82340-6516-43c2-b523-c40361c19560") : secret "dns-default-metrics-tls" not found
Sep 06 19:27:54.516623 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:54.516594    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls podName:c379232e-b6a5-4ba3-b941-1232bd22c1f3 nodeName:}" failed. No retries permitted until 2026-09-06 19:27:58.516588838 +0000 UTC m=+42.086887162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls") pod "image-registry-7df7c5cf9c-nd9t8" (UID: "c379232e-b6a5-4ba3-b941-1232bd22c1f3") : secret "image-registry-tls" not found
Sep 06 19:27:58.547339 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:58.547292    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:27:58.547693 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:27:58.547370    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:27:58.547693 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:58.547453    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 19:27:58.547693 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:58.547457    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:27:58.547693 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:58.547519    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls podName:7ef82340-6516-43c2-b523-c40361c19560 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:06.547503414 +0000 UTC m=+50.117801737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls") pod "dns-default-nfvjj" (UID: "7ef82340-6516-43c2-b523-c40361c19560") : secret "dns-default-metrics-tls" not found
Sep 06 19:27:58.547693 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:58.547465    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df7c5cf9c-nd9t8: secret "image-registry-tls" not found
Sep 06 19:27:58.547693 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:27:58.547587    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls podName:c379232e-b6a5-4ba3-b941-1232bd22c1f3 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:06.547576306 +0000 UTC m=+50.117874630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls") pod "image-registry-7df7c5cf9c-nd9t8" (UID: "c379232e-b6a5-4ba3-b941-1232bd22c1f3") : secret "image-registry-tls" not found
Sep 06 19:28:06.605412 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:06.605374    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:28:06.605867 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:06.605427    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:28:06.605867 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:06.605514    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:28:06.605867 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:06.605522    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 19:28:06.605867 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:06.605542    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df7c5cf9c-nd9t8: secret "image-registry-tls" not found
Sep 06 19:28:06.605867 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:06.605566    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls podName:7ef82340-6516-43c2-b523-c40361c19560 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:22.605551816 +0000 UTC m=+66.175850141 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls") pod "dns-default-nfvjj" (UID: "7ef82340-6516-43c2-b523-c40361c19560") : secret "dns-default-metrics-tls" not found
Sep 06 19:28:06.605867 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:06.605584    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls podName:c379232e-b6a5-4ba3-b941-1232bd22c1f3 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:22.605571726 +0000 UTC m=+66.175870084 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls") pod "image-registry-7df7c5cf9c-nd9t8" (UID: "c379232e-b6a5-4ba3-b941-1232bd22c1f3") : secret "image-registry-tls" not found
Sep 06 19:28:17.114897 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:17.114862    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n58nm"
Sep 06 19:28:21.416487 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.416451    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"]
Sep 06 19:28:21.445098 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.445071    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"]
Sep 06 19:28:21.445247 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.445187    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"
Sep 06 19:28:21.447590 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.447567    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 06 19:28:21.447590 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.447583    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 06 19:28:21.448539 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.448515    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 06 19:28:21.448688 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.448609    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vsd5q\""
Sep 06 19:28:21.514064 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.514025    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stqlt\" (UniqueName: \"kubernetes.io/projected/45ddfd90-02d0-4fb4-8d86-f1960f01362c-kube-api-access-stqlt\") pod \"cluster-samples-operator-7cdcfcf76c-flbj8\" (UID: \"45ddfd90-02d0-4fb4-8d86-f1960f01362c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"
Sep 06 19:28:21.514239 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.514090    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-flbj8\" (UID: \"45ddfd90-02d0-4fb4-8d86-f1960f01362c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"
Sep 06 19:28:21.607624 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.607586    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-bngcs"]
Sep 06 19:28:21.614949 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.614921    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stqlt\" (UniqueName: \"kubernetes.io/projected/45ddfd90-02d0-4fb4-8d86-f1960f01362c-kube-api-access-stqlt\") pod \"cluster-samples-operator-7cdcfcf76c-flbj8\" (UID: \"45ddfd90-02d0-4fb4-8d86-f1960f01362c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"
Sep 06 19:28:21.615038 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.614996    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-flbj8\" (UID: \"45ddfd90-02d0-4fb4-8d86-f1960f01362c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"
Sep 06 19:28:21.615122 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:21.615110    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 06 19:28:21.615189 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:21.615180    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls podName:45ddfd90-02d0-4fb4-8d86-f1960f01362c nodeName:}" failed. No retries permitted until 2026-09-06 19:28:22.115165865 +0000 UTC m=+65.685464189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-flbj8" (UID: "45ddfd90-02d0-4fb4-8d86-f1960f01362c") : secret "samples-operator-tls" not found
Sep 06 19:28:21.630912 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.630876    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stqlt\" (UniqueName: \"kubernetes.io/projected/45ddfd90-02d0-4fb4-8d86-f1960f01362c-kube-api-access-stqlt\") pod \"cluster-samples-operator-7cdcfcf76c-flbj8\" (UID: \"45ddfd90-02d0-4fb4-8d86-f1960f01362c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"
Sep 06 19:28:21.631077 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.631009    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-bngcs"]
Sep 06 19:28:21.631142 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.631116    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:21.633551 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.633524    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 06 19:28:21.633686 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.633554    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 06 19:28:21.633686 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.633607    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 06 19:28:21.633851 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.633837    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-k2knl\""
Sep 06 19:28:21.633942 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.633925    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 06 19:28:21.638313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.638290    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 06 19:28:21.716228 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.716129    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87c7f616-db47-42bd-b106-8231fff3aa41-config\") pod \"console-operator-85b454874b-bngcs\" (UID: \"87c7f616-db47-42bd-b106-8231fff3aa41\") " pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:21.716228 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.716188    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87c7f616-db47-42bd-b106-8231fff3aa41-trusted-ca\") pod \"console-operator-85b454874b-bngcs\" (UID: \"87c7f616-db47-42bd-b106-8231fff3aa41\") " pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:21.716228 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.716225    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9gk5\" (UniqueName: \"kubernetes.io/projected/87c7f616-db47-42bd-b106-8231fff3aa41-kube-api-access-k9gk5\") pod \"console-operator-85b454874b-bngcs\" (UID: \"87c7f616-db47-42bd-b106-8231fff3aa41\") " pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:21.716479 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.716327    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/87c7f616-db47-42bd-b106-8231fff3aa41-serving-cert\") pod \"console-operator-85b454874b-bngcs\" (UID: \"87c7f616-db47-42bd-b106-8231fff3aa41\") " pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:21.817098 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.817065    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87c7f616-db47-42bd-b106-8231fff3aa41-trusted-ca\") pod \"console-operator-85b454874b-bngcs\" (UID: \"87c7f616-db47-42bd-b106-8231fff3aa41\") " pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:21.817189 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.817114    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9gk5\" (UniqueName: \"kubernetes.io/projected/87c7f616-db47-42bd-b106-8231fff3aa41-kube-api-access-k9gk5\") pod \"console-operator-85b454874b-bngcs\" (UID: \"87c7f616-db47-42bd-b106-8231fff3aa41\") " pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:21.817189 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.817164    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/87c7f616-db47-42bd-b106-8231fff3aa41-serving-cert\") pod \"console-operator-85b454874b-bngcs\" (UID: \"87c7f616-db47-42bd-b106-8231fff3aa41\") " pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:21.817300 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.817198    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87c7f616-db47-42bd-b106-8231fff3aa41-config\") pod \"console-operator-85b454874b-bngcs\" (UID: \"87c7f616-db47-42bd-b106-8231fff3aa41\") " pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:21.817855 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.817829    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/87c7f616-db47-42bd-b106-8231fff3aa41-config\") pod \"console-operator-85b454874b-bngcs\" (UID: \"87c7f616-db47-42bd-b106-8231fff3aa41\") " pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:21.818025 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.818003    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87c7f616-db47-42bd-b106-8231fff3aa41-trusted-ca\") pod \"console-operator-85b454874b-bngcs\" (UID: \"87c7f616-db47-42bd-b106-8231fff3aa41\") " pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:21.819406 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.819389    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/87c7f616-db47-42bd-b106-8231fff3aa41-serving-cert\") pod \"console-operator-85b454874b-bngcs\" (UID: \"87c7f616-db47-42bd-b106-8231fff3aa41\") " pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:21.824880 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.824855    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9gk5\" (UniqueName: \"kubernetes.io/projected/87c7f616-db47-42bd-b106-8231fff3aa41-kube-api-access-k9gk5\") pod \"console-operator-85b454874b-bngcs\" (UID: \"87c7f616-db47-42bd-b106-8231fff3aa41\") " pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:21.943175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:21.943139    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:22.119337 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:22.119300    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-flbj8\" (UID: \"45ddfd90-02d0-4fb4-8d86-f1960f01362c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"
Sep 06 19:28:22.119521 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:22.119451    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 06 19:28:22.119521 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:22.119518    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls podName:45ddfd90-02d0-4fb4-8d86-f1960f01362c nodeName:}" failed. No retries permitted until 2026-09-06 19:28:23.119502644 +0000 UTC m=+66.689800986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-flbj8" (UID: "45ddfd90-02d0-4fb4-8d86-f1960f01362c") : secret "samples-operator-tls" not found
Sep 06 19:28:22.212466 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:22.212434    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-bngcs"]
Sep 06 19:28:22.215407 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:28:22.215379    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c7f616_db47_42bd_b106_8231fff3aa41.slice/crio-31d42eaf9d3ef53c471a60337c0eda1833f2a5311b88f301b30a7e796b3ab689 WatchSource:0}: Error finding container 31d42eaf9d3ef53c471a60337c0eda1833f2a5311b88f301b30a7e796b3ab689: Status 404 returned error can't find the container with id 31d42eaf9d3ef53c471a60337c0eda1833f2a5311b88f301b30a7e796b3ab689
Sep 06 19:28:22.623392 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:22.623341    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:28:22.623392 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:22.623392    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:28:22.623984 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:22.623418    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs\") pod \"network-metrics-daemon-pghst\" (UID: \"abf4350f-c25e-43ae-9271-5987625590f7\") " pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:28:22.623984 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:22.623443    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:28:22.623984 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:22.623524    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 19:28:22.623984 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:22.623548    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7df7c5cf9c-nd9t8: secret "image-registry-tls" not found
Sep 06 19:28:22.623984 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:22.623575    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 19:28:22.623984 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:22.623600    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls podName:c379232e-b6a5-4ba3-b941-1232bd22c1f3 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:54.623581907 +0000 UTC m=+98.193880233 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls") pod "image-registry-7df7c5cf9c-nd9t8" (UID: "c379232e-b6a5-4ba3-b941-1232bd22c1f3") : secret "image-registry-tls" not found
Sep 06 19:28:22.623984 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:22.623622    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls podName:7ef82340-6516-43c2-b523-c40361c19560 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:54.62360791 +0000 UTC m=+98.193906237 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls") pod "dns-default-nfvjj" (UID: "7ef82340-6516-43c2-b523-c40361c19560") : secret "dns-default-metrics-tls" not found
Sep 06 19:28:22.626194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:22.626174    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 06 19:28:22.626285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:22.626176    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 06 19:28:22.634271 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:22.634248    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 06 19:28:22.634413 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:22.634315    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs podName:abf4350f-c25e-43ae-9271-5987625590f7 nodeName:}" failed. No retries permitted until 2026-09-06 19:29:26.634295837 +0000 UTC m=+130.204594160 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs") pod "network-metrics-daemon-pghst" (UID: "abf4350f-c25e-43ae-9271-5987625590f7") : secret "metrics-daemon-secret" not found
Sep 06 19:28:22.636476 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:22.636456    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5e01f7a-8ff8-4821-b662-aed4b0d11377-original-pull-secret\") pod \"global-pull-secret-syncer-mp9fn\" (UID: \"e5e01f7a-8ff8-4821-b662-aed4b0d11377\") " pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:28:22.724459 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:22.724418    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdgt\" (UniqueName: \"kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt\") pod \"network-check-target-5b6qn\" (UID: \"11680e74-f55f-4749-a0a2-1180c6595524\") " pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:28:22.726882 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:22.726853    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 06 19:28:22.731703 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:22.731673    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp9fn"
Sep 06 19:28:22.737642 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:22.737609    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 06 19:28:22.749808 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:22.749780    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sdgt\" (UniqueName: \"kubernetes.io/projected/11680e74-f55f-4749-a0a2-1180c6595524-kube-api-access-9sdgt\") pod \"network-check-target-5b6qn\" (UID: \"11680e74-f55f-4749-a0a2-1180c6595524\") " pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:28:23.006702 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:23.006592    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mp9fn"]
Sep 06 19:28:23.010816 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:28:23.010781    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e01f7a_8ff8_4821_b662_aed4b0d11377.slice/crio-4cd2e7d8a643cf0dcb24eb72512a70937b76fdd45e457be72d2545fb5695b6e0 WatchSource:0}: Error finding container 4cd2e7d8a643cf0dcb24eb72512a70937b76fdd45e457be72d2545fb5695b6e0: Status 404 returned error can't find the container with id 4cd2e7d8a643cf0dcb24eb72512a70937b76fdd45e457be72d2545fb5695b6e0
Sep 06 19:28:23.039849 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:23.039813    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-26tb5\""
Sep 06 19:28:23.047706 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:23.047669    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:28:23.127580 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:23.127545    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-flbj8\" (UID: \"45ddfd90-02d0-4fb4-8d86-f1960f01362c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"
Sep 06 19:28:23.127745 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:23.127684    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 06 19:28:23.127819 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:23.127765    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls podName:45ddfd90-02d0-4fb4-8d86-f1960f01362c nodeName:}" failed. No retries permitted until 2026-09-06 19:28:25.12774429 +0000 UTC m=+68.698042631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-flbj8" (UID: "45ddfd90-02d0-4fb4-8d86-f1960f01362c") : secret "samples-operator-tls" not found
Sep 06 19:28:23.206393 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:23.206357    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mp9fn" event={"ID":"e5e01f7a-8ff8-4821-b662-aed4b0d11377","Type":"ContainerStarted","Data":"4cd2e7d8a643cf0dcb24eb72512a70937b76fdd45e457be72d2545fb5695b6e0"}
Sep 06 19:28:23.207676 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:23.207650    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-bngcs" event={"ID":"87c7f616-db47-42bd-b106-8231fff3aa41","Type":"ContainerStarted","Data":"31d42eaf9d3ef53c471a60337c0eda1833f2a5311b88f301b30a7e796b3ab689"}
Sep 06 19:28:23.322877 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:23.322838    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5b6qn"]
Sep 06 19:28:23.326306 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:28:23.326271    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11680e74_f55f_4749_a0a2_1180c6595524.slice/crio-5927f611aafc56c2c9cc543716103a669de041cf1733ea8ebd69649222afce87 WatchSource:0}: Error finding container 5927f611aafc56c2c9cc543716103a669de041cf1733ea8ebd69649222afce87: Status 404 returned error can't find the container with id 5927f611aafc56c2c9cc543716103a669de041cf1733ea8ebd69649222afce87
Sep 06 19:28:24.216279 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:24.215081    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-bngcs" event={"ID":"87c7f616-db47-42bd-b106-8231fff3aa41","Type":"ContainerStarted","Data":"6a63d777b44b28396b0e4c876c720d9b8a7e7c16d9b1e9fa865c45bce4d015ad"}
Sep 06 19:28:24.216279 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:24.216164    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:24.217812 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:24.217618    2571 patch_prober.go:28] interesting pod/console-operator-85b454874b-bngcs container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body=
Sep 06 19:28:24.217812 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:24.217676    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-85b454874b-bngcs" podUID="87c7f616-db47-42bd-b106-8231fff3aa41" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused"
Sep 06 19:28:24.218173 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:24.218143    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5b6qn" event={"ID":"11680e74-f55f-4749-a0a2-1180c6595524","Type":"ContainerStarted","Data":"5927f611aafc56c2c9cc543716103a669de041cf1733ea8ebd69649222afce87"}
Sep 06 19:28:24.232135 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:24.232078    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-bngcs" podStartSLOduration=1.381170156 podStartE2EDuration="3.23205538s" totalImagesPullingTime="1.850885224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:28:24.230686429 +0000 UTC m=+67.800984777" watchObservedRunningTime="2026-09-06 19:28:24.23205538 +0000 UTC m=+67.802353722"
Sep 06 19:28:25.144582 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:25.144523    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-flbj8\" (UID: \"45ddfd90-02d0-4fb4-8d86-f1960f01362c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"
Sep 06 19:28:25.144787 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:25.144681    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 06 19:28:25.144787 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:25.144755    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls podName:45ddfd90-02d0-4fb4-8d86-f1960f01362c nodeName:}" failed. No retries permitted until 2026-09-06 19:28:29.14473504 +0000 UTC m=+72.715033379 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-flbj8" (UID: "45ddfd90-02d0-4fb4-8d86-f1960f01362c") : secret "samples-operator-tls" not found
Sep 06 19:28:25.222794 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:25.222753    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/0.log"
Sep 06 19:28:25.223284 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:25.222799    2571 generic.go:356] "Generic (PLEG): container finished" podID="87c7f616-db47-42bd-b106-8231fff3aa41" containerID="6a63d777b44b28396b0e4c876c720d9b8a7e7c16d9b1e9fa865c45bce4d015ad" exitCode=255
Sep 06 19:28:25.223284 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:25.222880    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-bngcs" event={"ID":"87c7f616-db47-42bd-b106-8231fff3aa41","Type":"ContainerDied","Data":"6a63d777b44b28396b0e4c876c720d9b8a7e7c16d9b1e9fa865c45bce4d015ad"}
Sep 06 19:28:25.223284 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:25.223183    2571 scope.go:117] "RemoveContainer" containerID="6a63d777b44b28396b0e4c876c720d9b8a7e7c16d9b1e9fa865c45bce4d015ad"
Sep 06 19:28:26.227479 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:26.227446    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 19:28:26.227953 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:26.227922    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/0.log"
Sep 06 19:28:26.228044 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:26.227974    2571 generic.go:356] "Generic (PLEG): container finished" podID="87c7f616-db47-42bd-b106-8231fff3aa41" containerID="c3648ea8259bb33c095c0708e890fdb5e1b0e2454502d8bdecc328a8ca4fe20d" exitCode=255
Sep 06 19:28:26.228044 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:26.228022    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-bngcs" event={"ID":"87c7f616-db47-42bd-b106-8231fff3aa41","Type":"ContainerDied","Data":"c3648ea8259bb33c095c0708e890fdb5e1b0e2454502d8bdecc328a8ca4fe20d"}
Sep 06 19:28:26.228157 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:26.228075    2571 scope.go:117] "RemoveContainer" containerID="6a63d777b44b28396b0e4c876c720d9b8a7e7c16d9b1e9fa865c45bce4d015ad"
Sep 06 19:28:26.228333 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:26.228313    2571 scope.go:117] "RemoveContainer" containerID="c3648ea8259bb33c095c0708e890fdb5e1b0e2454502d8bdecc328a8ca4fe20d"
Sep 06 19:28:26.228546 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:26.228527    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-bngcs_openshift-console-operator(87c7f616-db47-42bd-b106-8231fff3aa41)\"" pod="openshift-console-operator/console-operator-85b454874b-bngcs" podUID="87c7f616-db47-42bd-b106-8231fff3aa41"
Sep 06 19:28:27.230868 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.230830    2571 scope.go:117] "RemoveContainer" containerID="c3648ea8259bb33c095c0708e890fdb5e1b0e2454502d8bdecc328a8ca4fe20d"
Sep 06 19:28:27.231361 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:27.231028    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-bngcs_openshift-console-operator(87c7f616-db47-42bd-b106-8231fff3aa41)\"" pod="openshift-console-operator/console-operator-85b454874b-bngcs" podUID="87c7f616-db47-42bd-b106-8231fff3aa41"
Sep 06 19:28:27.306616 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.306578    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vj2gh"]
Sep 06 19:28:27.329118 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.329078    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vj2gh"]
Sep 06 19:28:27.329279 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.329208    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.331888 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.331851    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 06 19:28:27.332064 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.331894    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 06 19:28:27.332064 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.331917    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 06 19:28:27.332200 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.332184    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8jvvj\""
Sep 06 19:28:27.332200 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.332191    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 06 19:28:27.364693 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.364655    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96c6d\" (UniqueName: \"kubernetes.io/projected/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-api-access-96c6d\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.364893 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.364723    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.364893 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.364826    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.364893 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.364884    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/144dc9aa-3055-4442-9f9b-c9290a2d48e3-crio-socket\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.365069 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.364919    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/144dc9aa-3055-4442-9f9b-c9290a2d48e3-data-volume\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.465533 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.465494    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.465730 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.465554    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.465730 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.465600    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/144dc9aa-3055-4442-9f9b-c9290a2d48e3-crio-socket\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.465730 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:27.465616    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 19:28:27.465730 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.465624    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/144dc9aa-3055-4442-9f9b-c9290a2d48e3-data-volume\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.465730 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:27.465665    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 19:28:27.465730 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:27.465694    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm podName:144dc9aa-3055-4442-9f9b-c9290a2d48e3 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:27.965672973 +0000 UTC m=+71.535971313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vj2gh" (UID: "144dc9aa-3055-4442-9f9b-c9290a2d48e3") : configmap "kube-rbac-proxy" not found
Sep 06 19:28:27.465730 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:27.465728    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls podName:144dc9aa-3055-4442-9f9b-c9290a2d48e3 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:27.965710198 +0000 UTC m=+71.536008533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vj2gh" (UID: "144dc9aa-3055-4442-9f9b-c9290a2d48e3") : secret "insights-runtime-extractor-tls" not found
Sep 06 19:28:27.466132 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.465736    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/144dc9aa-3055-4442-9f9b-c9290a2d48e3-crio-socket\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.466132 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.465749    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96c6d\" (UniqueName: \"kubernetes.io/projected/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-api-access-96c6d\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.466132 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.466029    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/144dc9aa-3055-4442-9f9b-c9290a2d48e3-data-volume\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.474110 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.474078    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96c6d\" (UniqueName: \"kubernetes.io/projected/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-api-access-96c6d\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.969433 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.969394    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.969577 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:27.969448    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:27.969577 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:27.969517    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 19:28:27.969577 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:27.969548    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 19:28:27.969696 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:27.969585    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm podName:144dc9aa-3055-4442-9f9b-c9290a2d48e3 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:28.969568969 +0000 UTC m=+72.539867293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vj2gh" (UID: "144dc9aa-3055-4442-9f9b-c9290a2d48e3") : configmap "kube-rbac-proxy" not found
Sep 06 19:28:27.969696 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:27.969600    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls podName:144dc9aa-3055-4442-9f9b-c9290a2d48e3 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:28.969593829 +0000 UTC m=+72.539892153 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vj2gh" (UID: "144dc9aa-3055-4442-9f9b-c9290a2d48e3") : secret "insights-runtime-extractor-tls" not found
Sep 06 19:28:28.235556 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.235443    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mp9fn" event={"ID":"e5e01f7a-8ff8-4821-b662-aed4b0d11377","Type":"ContainerStarted","Data":"488a5faf16f3b940e76bde60d121fa4d33307ccd94642655b273454781791747"}
Sep 06 19:28:28.237123 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.237087    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5b6qn" event={"ID":"11680e74-f55f-4749-a0a2-1180c6595524","Type":"ContainerStarted","Data":"e0bfe09cfbf6981a2bc9c3bb46c04d4bd6612d8318893b8b2a4aade40dafcc0f"}
Sep 06 19:28:28.237266 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.237154    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:28:28.238648 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.238631    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 19:28:28.249844 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.249781    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mp9fn" podStartSLOduration=66.239647398 podStartE2EDuration="1m11.249759457s" totalImagesPullingTime="5.010112059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:28:28.248361401 +0000 UTC m=+71.818659747" watchObservedRunningTime="2026-09-06 19:28:28.249759457 +0000 UTC m=+71.820057804"
Sep 06 19:28:28.261738 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.261686    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5b6qn" podStartSLOduration=66.571687748 podStartE2EDuration="1m11.261670394s" totalImagesPullingTime="4.689982646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:28:28.260913368 +0000 UTC m=+71.831211715" watchObservedRunningTime="2026-09-06 19:28:28.261670394 +0000 UTC m=+71.831968739"
Sep 06 19:28:28.582069 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.582031    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-vz6km"]
Sep 06 19:28:28.584996 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.584979    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-vz6km"
Sep 06 19:28:28.587451 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.587414    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 06 19:28:28.587451 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.587443    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 06 19:28:28.587651 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.587456    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 06 19:28:28.587651 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.587543    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jrsvg\""
Sep 06 19:28:28.587651 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.587580    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 06 19:28:28.593315 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.593295    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-vz6km"]
Sep 06 19:28:28.675499 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.675462    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc-signing-key\") pod \"service-ca-75d99c484-vz6km\" (UID: \"f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc\") " pod="openshift-service-ca/service-ca-75d99c484-vz6km"
Sep 06 19:28:28.675499 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.675516    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc-signing-cabundle\") pod \"service-ca-75d99c484-vz6km\" (UID: \"f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc\") " pod="openshift-service-ca/service-ca-75d99c484-vz6km"
Sep 06 19:28:28.675718 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.675636    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv28r\" (UniqueName: \"kubernetes.io/projected/f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc-kube-api-access-cv28r\") pod \"service-ca-75d99c484-vz6km\" (UID: \"f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc\") " pod="openshift-service-ca/service-ca-75d99c484-vz6km"
Sep 06 19:28:28.776451 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.776412    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc-signing-key\") pod \"service-ca-75d99c484-vz6km\" (UID: \"f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc\") " pod="openshift-service-ca/service-ca-75d99c484-vz6km"
Sep 06 19:28:28.776451 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.776468    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc-signing-cabundle\") pod \"service-ca-75d99c484-vz6km\" (UID: \"f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc\") " pod="openshift-service-ca/service-ca-75d99c484-vz6km"
Sep 06 19:28:28.776718 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.776563    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv28r\" (UniqueName: \"kubernetes.io/projected/f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc-kube-api-access-cv28r\") pod \"service-ca-75d99c484-vz6km\" (UID: \"f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc\") " pod="openshift-service-ca/service-ca-75d99c484-vz6km"
Sep 06 19:28:28.777224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.777200    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc-signing-cabundle\") pod \"service-ca-75d99c484-vz6km\" (UID: \"f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc\") " pod="openshift-service-ca/service-ca-75d99c484-vz6km"
Sep 06 19:28:28.778916 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.778894    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc-signing-key\") pod \"service-ca-75d99c484-vz6km\" (UID: \"f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc\") " pod="openshift-service-ca/service-ca-75d99c484-vz6km"
Sep 06 19:28:28.784313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.784287    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv28r\" (UniqueName: \"kubernetes.io/projected/f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc-kube-api-access-cv28r\") pod \"service-ca-75d99c484-vz6km\" (UID: \"f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc\") " pod="openshift-service-ca/service-ca-75d99c484-vz6km"
Sep 06 19:28:28.894135 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.894050    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-vz6km"
Sep 06 19:28:28.978737 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.978697    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:28.978885 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:28.978758    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:28.978885 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:28.978840    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 19:28:28.978984 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:28.978912    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm podName:144dc9aa-3055-4442-9f9b-c9290a2d48e3 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:30.978895508 +0000 UTC m=+74.549193836 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vj2gh" (UID: "144dc9aa-3055-4442-9f9b-c9290a2d48e3") : configmap "kube-rbac-proxy" not found
Sep 06 19:28:28.978984 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:28.978912    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 19:28:28.978984 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:28.978978    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls podName:144dc9aa-3055-4442-9f9b-c9290a2d48e3 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:30.97895067 +0000 UTC m=+74.549248993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vj2gh" (UID: "144dc9aa-3055-4442-9f9b-c9290a2d48e3") : secret "insights-runtime-extractor-tls" not found
Sep 06 19:28:29.090238 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:29.090210    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fs4k9_684b35b0-78f1-4eb5-9d65-2a5b5caf6464/dns-node-resolver/0.log"
Sep 06 19:28:29.168028 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:29.167925    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-vz6km"]
Sep 06 19:28:29.171058 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:28:29.171021    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf283ed6d_4adf_4b7d_ae2c_59084ba1d8dc.slice/crio-1a25e90e53d1584dca56e5bd600d9755d3774e4eb675ed53faa3c2c76c489bbb WatchSource:0}: Error finding container 1a25e90e53d1584dca56e5bd600d9755d3774e4eb675ed53faa3c2c76c489bbb: Status 404 returned error can't find the container with id 1a25e90e53d1584dca56e5bd600d9755d3774e4eb675ed53faa3c2c76c489bbb
Sep 06 19:28:29.180989 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:29.180945    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-flbj8\" (UID: \"45ddfd90-02d0-4fb4-8d86-f1960f01362c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"
Sep 06 19:28:29.181081 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:29.181058    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 06 19:28:29.181120 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:29.181114    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls podName:45ddfd90-02d0-4fb4-8d86-f1960f01362c nodeName:}" failed. No retries permitted until 2026-09-06 19:28:37.18109972 +0000 UTC m=+80.751398043 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-flbj8" (UID: "45ddfd90-02d0-4fb4-8d86-f1960f01362c") : secret "samples-operator-tls" not found
Sep 06 19:28:29.242029 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:29.241993    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-vz6km" event={"ID":"f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc","Type":"ContainerStarted","Data":"1a25e90e53d1584dca56e5bd600d9755d3774e4eb675ed53faa3c2c76c489bbb"}
Sep 06 19:28:30.490016 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:30.489982    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z9s22_bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f/node-ca/0.log"
Sep 06 19:28:30.997777 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:30.997736    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:30.997987 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:30.997793    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:30.997987 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:30.997868    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 19:28:30.997987 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:30.997926    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 19:28:30.997987 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:30.997940    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm podName:144dc9aa-3055-4442-9f9b-c9290a2d48e3 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:34.997923535 +0000 UTC m=+78.568221881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vj2gh" (UID: "144dc9aa-3055-4442-9f9b-c9290a2d48e3") : configmap "kube-rbac-proxy" not found
Sep 06 19:28:30.997987 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:30.997986    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls podName:144dc9aa-3055-4442-9f9b-c9290a2d48e3 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:34.997952993 +0000 UTC m=+78.568251317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vj2gh" (UID: "144dc9aa-3055-4442-9f9b-c9290a2d48e3") : secret "insights-runtime-extractor-tls" not found
Sep 06 19:28:31.088706 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:31.088672    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sr895_f4039f06-ffbd-4400-941c-beb1104ea6f1/serve-healthcheck-canary/0.log"
Sep 06 19:28:31.251879 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:31.251784    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-vz6km" event={"ID":"f283ed6d-4adf-4b7d-ae2c-59084ba1d8dc","Type":"ContainerStarted","Data":"05fd2d0fa38e69843a5df1118e33a415245f4d6b0975543f72b19cc854e8e1bf"}
Sep 06 19:28:31.265253 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:31.265197    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-vz6km" podStartSLOduration=1.550149138 podStartE2EDuration="3.2651809s" totalImagesPullingTime="1.715031762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:28:31.264636837 +0000 UTC m=+74.834935183" watchObservedRunningTime="2026-09-06 19:28:31.2651809 +0000 UTC m=+74.835479245"
Sep 06 19:28:31.943521 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:31.943482    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:31.943915 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:31.943850    2571 scope.go:117] "RemoveContainer" containerID="c3648ea8259bb33c095c0708e890fdb5e1b0e2454502d8bdecc328a8ca4fe20d"
Sep 06 19:28:31.944032 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:31.944012    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-bngcs_openshift-console-operator(87c7f616-db47-42bd-b106-8231fff3aa41)\"" pod="openshift-console-operator/console-operator-85b454874b-bngcs" podUID="87c7f616-db47-42bd-b106-8231fff3aa41"
Sep 06 19:28:34.216634 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:34.216592    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:34.217267 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:34.217109    2571 scope.go:117] "RemoveContainer" containerID="c3648ea8259bb33c095c0708e890fdb5e1b0e2454502d8bdecc328a8ca4fe20d"
Sep 06 19:28:34.217345 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:34.217315    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-bngcs_openshift-console-operator(87c7f616-db47-42bd-b106-8231fff3aa41)\"" pod="openshift-console-operator/console-operator-85b454874b-bngcs" podUID="87c7f616-db47-42bd-b106-8231fff3aa41"
Sep 06 19:28:35.029245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:35.029200    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:35.029245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:35.029259    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:35.029530 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:35.029354    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 19:28:35.029530 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:35.029358    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 19:28:35.029530 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:35.029412    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls podName:144dc9aa-3055-4442-9f9b-c9290a2d48e3 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:43.029398481 +0000 UTC m=+86.599696810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vj2gh" (UID: "144dc9aa-3055-4442-9f9b-c9290a2d48e3") : secret "insights-runtime-extractor-tls" not found
Sep 06 19:28:35.029530 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:35.029427    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm podName:144dc9aa-3055-4442-9f9b-c9290a2d48e3 nodeName:}" failed. No retries permitted until 2026-09-06 19:28:43.029420442 +0000 UTC m=+86.599718766 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vj2gh" (UID: "144dc9aa-3055-4442-9f9b-c9290a2d48e3") : configmap "kube-rbac-proxy" not found
Sep 06 19:28:37.247457 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:37.247416    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-flbj8\" (UID: \"45ddfd90-02d0-4fb4-8d86-f1960f01362c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"
Sep 06 19:28:37.249814 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:37.249792    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/45ddfd90-02d0-4fb4-8d86-f1960f01362c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-flbj8\" (UID: \"45ddfd90-02d0-4fb4-8d86-f1960f01362c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"
Sep 06 19:28:37.353728 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:37.353682    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"
Sep 06 19:28:37.618813 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:37.618779    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8"]
Sep 06 19:28:38.270502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:38.270468    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8" event={"ID":"45ddfd90-02d0-4fb4-8d86-f1960f01362c","Type":"ContainerStarted","Data":"f8bb4c1268d19a607bed264b8b1b68dc912453b92e78443327642fd38d5169e7"}
Sep 06 19:28:40.277931 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:40.277894    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8" event={"ID":"45ddfd90-02d0-4fb4-8d86-f1960f01362c","Type":"ContainerStarted","Data":"e4b5581230e005eee1e2efeac4395c89cbf1ff3c5ad0f33affbebb081632b6fd"}
Sep 06 19:28:40.278429 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:40.278025    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8" event={"ID":"45ddfd90-02d0-4fb4-8d86-f1960f01362c","Type":"ContainerStarted","Data":"b7af544652f4e12122ead43f4214efa245169af685c8f09ac4e59e2275b119f8"}
Sep 06 19:28:40.292836 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:40.292778    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-flbj8" podStartSLOduration=17.512007089 podStartE2EDuration="19.292761272s" totalImagesPullingTime="1.780754183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:28:40.291553081 +0000 UTC m=+83.861851426" watchObservedRunningTime="2026-09-06 19:28:40.292761272 +0000 UTC m=+83.863059646"
Sep 06 19:28:43.100218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:43.100159    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:43.100218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:43.100223    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:43.100853 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:43.100829    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144dc9aa-3055-4442-9f9b-c9290a2d48e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:43.102520 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:43.102498    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144dc9aa-3055-4442-9f9b-c9290a2d48e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj2gh\" (UID: \"144dc9aa-3055-4442-9f9b-c9290a2d48e3\") " pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:43.240222 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:43.240179    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vj2gh"
Sep 06 19:28:43.519841 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:43.519747    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vj2gh"]
Sep 06 19:28:43.523375 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:28:43.523342    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144dc9aa_3055_4442_9f9b_c9290a2d48e3.slice/crio-691b8cc8768a96a7559e41a701fc0b702d359ac887d8addf3862e4106e61cc7a WatchSource:0}: Error finding container 691b8cc8768a96a7559e41a701fc0b702d359ac887d8addf3862e4106e61cc7a: Status 404 returned error can't find the container with id 691b8cc8768a96a7559e41a701fc0b702d359ac887d8addf3862e4106e61cc7a
Sep 06 19:28:44.295410 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:44.295376    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vj2gh" event={"ID":"144dc9aa-3055-4442-9f9b-c9290a2d48e3","Type":"ContainerStarted","Data":"e065055e98f952c73bc1c414f141a5b5b53ec90b4585f27a181aa86d76de4d27"}
Sep 06 19:28:44.295780 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:44.295461    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vj2gh" event={"ID":"144dc9aa-3055-4442-9f9b-c9290a2d48e3","Type":"ContainerStarted","Data":"7170602dd98e0d3e50bcde0a76b34ae5c0bb9576a0d0feed25a1ad2136cb30ec"}
Sep 06 19:28:44.295780 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:44.295483    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vj2gh" event={"ID":"144dc9aa-3055-4442-9f9b-c9290a2d48e3","Type":"ContainerStarted","Data":"691b8cc8768a96a7559e41a701fc0b702d359ac887d8addf3862e4106e61cc7a"}
Sep 06 19:28:46.304286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:46.304251    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vj2gh" event={"ID":"144dc9aa-3055-4442-9f9b-c9290a2d48e3","Type":"ContainerStarted","Data":"9db84aa994b9985ce888df1e0e96808d40e372be2270f7bf4c93b1099d1f44c6"}
Sep 06 19:28:46.320458 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:46.320407    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vj2gh" podStartSLOduration=17.398530185 podStartE2EDuration="19.32039152s" totalImagesPullingTime="1.921861335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 19:28:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:28:46.318939622 +0000 UTC m=+89.889237970" watchObservedRunningTime="2026-09-06 19:28:46.32039152 +0000 UTC m=+89.890689887"
Sep 06 19:28:48.126918 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.126885    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75746d974-q5gv5"]
Sep 06 19:28:48.130187 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.130164    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75746d974-q5gv5"
Sep 06 19:28:48.132796 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.132774    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 06 19:28:48.132891 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.132804    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 06 19:28:48.133665 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.133648    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-85krm\""
Sep 06 19:28:48.134038 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.134022    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 06 19:28:48.134122 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.134027    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 06 19:28:48.139676 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.139653    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75746d974-q5gv5"]
Sep 06 19:28:48.241209 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.241173    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwxqh\" (UniqueName: \"kubernetes.io/projected/61563462-0536-43e2-9dae-24bc8b3d93cb-kube-api-access-vwxqh\") pod \"managed-serviceaccount-addon-agent-75746d974-q5gv5\" (UID: \"61563462-0536-43e2-9dae-24bc8b3d93cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75746d974-q5gv5"
Sep 06 19:28:48.241377 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.241303    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61563462-0536-43e2-9dae-24bc8b3d93cb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75746d974-q5gv5\" (UID: \"61563462-0536-43e2-9dae-24bc8b3d93cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75746d974-q5gv5"
Sep 06 19:28:48.342231 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.342191    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61563462-0536-43e2-9dae-24bc8b3d93cb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75746d974-q5gv5\" (UID: \"61563462-0536-43e2-9dae-24bc8b3d93cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75746d974-q5gv5"
Sep 06 19:28:48.342392 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.342258    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxqh\" (UniqueName: \"kubernetes.io/projected/61563462-0536-43e2-9dae-24bc8b3d93cb-kube-api-access-vwxqh\") pod \"managed-serviceaccount-addon-agent-75746d974-q5gv5\" (UID: \"61563462-0536-43e2-9dae-24bc8b3d93cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75746d974-q5gv5"
Sep 06 19:28:48.344908 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.344878    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61563462-0536-43e2-9dae-24bc8b3d93cb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75746d974-q5gv5\" (UID: \"61563462-0536-43e2-9dae-24bc8b3d93cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75746d974-q5gv5"
Sep 06 19:28:48.358724 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.358693    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwxqh\" (UniqueName: \"kubernetes.io/projected/61563462-0536-43e2-9dae-24bc8b3d93cb-kube-api-access-vwxqh\") pod \"managed-serviceaccount-addon-agent-75746d974-q5gv5\" (UID: \"61563462-0536-43e2-9dae-24bc8b3d93cb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75746d974-q5gv5"
Sep 06 19:28:48.449570 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.449474    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75746d974-q5gv5"
Sep 06 19:28:48.732250 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:48.732168    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75746d974-q5gv5"]
Sep 06 19:28:48.735809 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:28:48.735781    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61563462_0536_43e2_9dae_24bc8b3d93cb.slice/crio-78750f6f5a27142bb60f042d6065774af3d23fcd95a8aaf6c547f773d63efd59 WatchSource:0}: Error finding container 78750f6f5a27142bb60f042d6065774af3d23fcd95a8aaf6c547f773d63efd59: Status 404 returned error can't find the container with id 78750f6f5a27142bb60f042d6065774af3d23fcd95a8aaf6c547f773d63efd59
Sep 06 19:28:49.313936 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:49.313901    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75746d974-q5gv5" event={"ID":"61563462-0536-43e2-9dae-24bc8b3d93cb","Type":"ContainerStarted","Data":"78750f6f5a27142bb60f042d6065774af3d23fcd95a8aaf6c547f773d63efd59"}
Sep 06 19:28:49.912602 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:49.912342    2571 scope.go:117] "RemoveContainer" containerID="c3648ea8259bb33c095c0708e890fdb5e1b0e2454502d8bdecc328a8ca4fe20d"
Sep 06 19:28:50.322387 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:50.322356    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 19:28:50.322835 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:50.322440    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-bngcs" event={"ID":"87c7f616-db47-42bd-b106-8231fff3aa41","Type":"ContainerStarted","Data":"1b9bad313fa37041b9f2e12f783e06b7cbeb1df5d56697afde5d62556574f0d9"}
Sep 06 19:28:50.322835 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:50.322815    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:51.040428 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:51.040392    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-bngcs"
Sep 06 19:28:51.212020 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:51.211981    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-wvkbc"]
Sep 06 19:28:51.215430 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:51.215405    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-wvkbc"
Sep 06 19:28:51.217989 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:51.217922    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 06 19:28:51.217989 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:51.217939    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-nqgpt\""
Sep 06 19:28:51.217989 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:51.217939    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 06 19:28:51.225246 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:51.225219    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-wvkbc"]
Sep 06 19:28:51.269142 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:51.269106    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44n67\" (UniqueName: \"kubernetes.io/projected/9557b720-35be-4861-8d3a-c82a3acf997e-kube-api-access-44n67\") pod \"downloads-85d98c998f-wvkbc\" (UID: \"9557b720-35be-4861-8d3a-c82a3acf997e\") " pod="openshift-console/downloads-85d98c998f-wvkbc"
Sep 06 19:28:51.370267 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:51.370234    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44n67\" (UniqueName: \"kubernetes.io/projected/9557b720-35be-4861-8d3a-c82a3acf997e-kube-api-access-44n67\") pod \"downloads-85d98c998f-wvkbc\" (UID: \"9557b720-35be-4861-8d3a-c82a3acf997e\") " pod="openshift-console/downloads-85d98c998f-wvkbc"
Sep 06 19:28:51.377538 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:51.377512    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44n67\" (UniqueName: \"kubernetes.io/projected/9557b720-35be-4861-8d3a-c82a3acf997e-kube-api-access-44n67\") pod \"downloads-85d98c998f-wvkbc\" (UID: \"9557b720-35be-4861-8d3a-c82a3acf997e\") " pod="openshift-console/downloads-85d98c998f-wvkbc"
Sep 06 19:28:51.527901 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:51.527869    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-wvkbc"
Sep 06 19:28:51.798923 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:51.798842    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-wvkbc"]
Sep 06 19:28:51.801689 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:28:51.801663    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9557b720_35be_4861_8d3a_c82a3acf997e.slice/crio-74eccba3534996becca171fc5bcf19cce9addd2c8c5bedfe8a4cba6cfb6a4441 WatchSource:0}: Error finding container 74eccba3534996becca171fc5bcf19cce9addd2c8c5bedfe8a4cba6cfb6a4441: Status 404 returned error can't find the container with id 74eccba3534996becca171fc5bcf19cce9addd2c8c5bedfe8a4cba6cfb6a4441
Sep 06 19:28:52.330119 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:52.330079    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75746d974-q5gv5" event={"ID":"61563462-0536-43e2-9dae-24bc8b3d93cb","Type":"ContainerStarted","Data":"9145017ca25d299e20408d65caa35a35b719ae2bc0cec90afa24ea505d274e25"}
Sep 06 19:28:52.331258 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:52.331235    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-wvkbc" event={"ID":"9557b720-35be-4861-8d3a-c82a3acf997e","Type":"ContainerStarted","Data":"74eccba3534996becca171fc5bcf19cce9addd2c8c5bedfe8a4cba6cfb6a4441"}
Sep 06 19:28:52.343428 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:52.343174    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75746d974-q5gv5" podStartSLOduration=1.690499771 podStartE2EDuration="4.343154731s" totalImagesPullingTime="2.65265496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:28:52.342497094 +0000 UTC m=+95.912795443" watchObservedRunningTime="2026-09-06 19:28:52.343154731 +0000 UTC m=+95.913453078"
Sep 06 19:28:54.697994 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:54.697907    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:28:54.698425 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:54.698003    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:28:54.700721 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:54.700689    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ef82340-6516-43c2-b523-c40361c19560-metrics-tls\") pod \"dns-default-nfvjj\" (UID: \"7ef82340-6516-43c2-b523-c40361c19560\") " pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:28:54.700851 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:54.700836    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls\") pod \"image-registry-7df7c5cf9c-nd9t8\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") " pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:28:54.973281 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:54.973200    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sn49m\""
Sep 06 19:28:54.982065 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:54.982029    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:28:54.985395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:54.985371    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kglk6\""
Sep 06 19:28:54.994277 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:54.994251    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:28:55.267134 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:55.267057    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"]
Sep 06 19:28:55.271284 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:28:55.271251    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc379232e_b6a5_4ba3_b941_1232bd22c1f3.slice/crio-c10fdee18deecd35d41f7fa0d1c466942bae171b5333751f7c4ed879dbd29c04 WatchSource:0}: Error finding container c10fdee18deecd35d41f7fa0d1c466942bae171b5333751f7c4ed879dbd29c04: Status 404 returned error can't find the container with id c10fdee18deecd35d41f7fa0d1c466942bae171b5333751f7c4ed879dbd29c04
Sep 06 19:28:55.284937 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:55.284877    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nfvjj"]
Sep 06 19:28:55.289620 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:28:55.289581    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef82340_6516_43c2_b523_c40361c19560.slice/crio-1ed703e197c5b6701b090f94375f484fbb1a8a68829c129272c2324a1aac5644 WatchSource:0}: Error finding container 1ed703e197c5b6701b090f94375f484fbb1a8a68829c129272c2324a1aac5644: Status 404 returned error can't find the container with id 1ed703e197c5b6701b090f94375f484fbb1a8a68829c129272c2324a1aac5644
Sep 06 19:28:55.343022 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:55.342991    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nfvjj" event={"ID":"7ef82340-6516-43c2-b523-c40361c19560","Type":"ContainerStarted","Data":"1ed703e197c5b6701b090f94375f484fbb1a8a68829c129272c2324a1aac5644"}
Sep 06 19:28:55.344863 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:55.344831    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8" event={"ID":"c379232e-b6a5-4ba3-b941-1232bd22c1f3","Type":"ContainerStarted","Data":"c10fdee18deecd35d41f7fa0d1c466942bae171b5333751f7c4ed879dbd29c04"}
Sep 06 19:28:55.939750 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:55.939700    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qrrht"]
Sep 06 19:28:55.944838 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:55.944805    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qrrht"
Sep 06 19:28:55.947261 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:55.947233    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 06 19:28:55.947395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:55.947340    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-wd9xp\""
Sep 06 19:28:55.950692 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:55.950667    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qrrht"]
Sep 06 19:28:56.009388 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:56.009275    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f8b821a1-8324-47a9-afda-fc17281c03df-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-qrrht\" (UID: \"f8b821a1-8324-47a9-afda-fc17281c03df\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qrrht"
Sep 06 19:28:56.110742 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:56.110697    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f8b821a1-8324-47a9-afda-fc17281c03df-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-qrrht\" (UID: \"f8b821a1-8324-47a9-afda-fc17281c03df\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qrrht"
Sep 06 19:28:56.110925 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:56.110902    2571 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 06 19:28:56.111027 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:28:56.110985    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8b821a1-8324-47a9-afda-fc17281c03df-tls-certificates podName:f8b821a1-8324-47a9-afda-fc17281c03df nodeName:}" failed. No retries permitted until 2026-09-06 19:28:56.610946099 +0000 UTC m=+100.181244429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/f8b821a1-8324-47a9-afda-fc17281c03df-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-qrrht" (UID: "f8b821a1-8324-47a9-afda-fc17281c03df") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 06 19:28:56.351564 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:56.351527    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8" event={"ID":"c379232e-b6a5-4ba3-b941-1232bd22c1f3","Type":"ContainerStarted","Data":"755a0047014d0b894afec9485e684b0aaa6fbca9b47e6876a1d03b110bf5333b"}
Sep 06 19:28:56.351755 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:56.351619    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:28:56.370595 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:56.370530    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8" podStartSLOduration=99.370505373 podStartE2EDuration="1m39.370505373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:28:56.369103317 +0000 UTC m=+99.939401663" watchObservedRunningTime="2026-09-06 19:28:56.370505373 +0000 UTC m=+99.940803721"
Sep 06 19:28:56.615566 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:56.615474    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f8b821a1-8324-47a9-afda-fc17281c03df-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-qrrht\" (UID: \"f8b821a1-8324-47a9-afda-fc17281c03df\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qrrht"
Sep 06 19:28:56.618674 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:56.618615    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f8b821a1-8324-47a9-afda-fc17281c03df-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-qrrht\" (UID: \"f8b821a1-8324-47a9-afda-fc17281c03df\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qrrht"
Sep 06 19:28:56.860297 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:56.860130    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qrrht"
Sep 06 19:28:57.260271 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:57.260163    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qrrht"]
Sep 06 19:28:57.263650 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:28:57.263614    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b821a1_8324_47a9_afda_fc17281c03df.slice/crio-fe909327cbcd7013f2c475f8ffccf7976097be747994488a51b4103234dc01c6 WatchSource:0}: Error finding container fe909327cbcd7013f2c475f8ffccf7976097be747994488a51b4103234dc01c6: Status 404 returned error can't find the container with id fe909327cbcd7013f2c475f8ffccf7976097be747994488a51b4103234dc01c6
Sep 06 19:28:57.358012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:57.357952    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nfvjj" event={"ID":"7ef82340-6516-43c2-b523-c40361c19560","Type":"ContainerStarted","Data":"4b293f17d918d08b718120c172690b1a86a9e6ef5b1034b98955ec44b5351ecc"}
Sep 06 19:28:57.358188 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:57.358038    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nfvjj" event={"ID":"7ef82340-6516-43c2-b523-c40361c19560","Type":"ContainerStarted","Data":"30acbbf89cfc1c43b4e45fa91dddd4a96905eab07c77f23d3fdac5bb8f088a87"}
Sep 06 19:28:57.358188 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:57.358090    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:28:57.359647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:57.359580    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qrrht" event={"ID":"f8b821a1-8324-47a9-afda-fc17281c03df","Type":"ContainerStarted","Data":"fe909327cbcd7013f2c475f8ffccf7976097be747994488a51b4103234dc01c6"}
Sep 06 19:28:57.374102 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:57.374049    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nfvjj" podStartSLOduration=65.626303506 podStartE2EDuration="1m7.374034266s" totalImagesPullingTime="1.74773076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:28:57.372324074 +0000 UTC m=+100.942622423" watchObservedRunningTime="2026-09-06 19:28:57.374034266 +0000 UTC m=+100.944332612"
Sep 06 19:28:59.245431 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:59.245402    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5b6qn"
Sep 06 19:28:59.370182 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:59.370146    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qrrht" event={"ID":"f8b821a1-8324-47a9-afda-fc17281c03df","Type":"ContainerStarted","Data":"701e590a16bb6cac82c0b3431d1cf9b62a0ef2783686fb5e884d8368c4849844"}
Sep 06 19:28:59.370395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:59.370375    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qrrht"
Sep 06 19:28:59.375998 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:59.375950    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qrrht"
Sep 06 19:28:59.383463 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:59.383419    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-qrrht" podStartSLOduration=3.14000372 podStartE2EDuration="4.383400048s" totalImagesPullingTime="1.243396328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:28:59.383196188 +0000 UTC m=+102.953494534" watchObservedRunningTime="2026-09-06 19:28:59.383400048 +0000 UTC m=+102.953698397"
Sep 06 19:28:59.993350 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:28:59.993313    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"]
Sep 06 19:29:00.008061 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.008028    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"]
Sep 06 19:29:00.008230 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.008174    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"
Sep 06 19:29:00.010477 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.010353    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 06 19:29:00.011861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.011594    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4pp6l\""
Sep 06 19:29:00.011861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.011614    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 06 19:29:00.011861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.011670    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 06 19:29:00.011861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.011731    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 06 19:29:00.011861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.011792    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 06 19:29:00.147329 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.147291    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2587a67-5897-4d57-b72b-7cfcf18b6377-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-rw5l6\" (UID: \"b2587a67-5897-4d57-b72b-7cfcf18b6377\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"
Sep 06 19:29:00.147329 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.147333    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2587a67-5897-4d57-b72b-7cfcf18b6377-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-rw5l6\" (UID: \"b2587a67-5897-4d57-b72b-7cfcf18b6377\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"
Sep 06 19:29:00.147578 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.147367    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8r5k\" (UniqueName: \"kubernetes.io/projected/b2587a67-5897-4d57-b72b-7cfcf18b6377-kube-api-access-t8r5k\") pod \"prometheus-operator-6f4946db56-rw5l6\" (UID: \"b2587a67-5897-4d57-b72b-7cfcf18b6377\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"
Sep 06 19:29:00.147578 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.147393    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2587a67-5897-4d57-b72b-7cfcf18b6377-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-rw5l6\" (UID: \"b2587a67-5897-4d57-b72b-7cfcf18b6377\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"
Sep 06 19:29:00.248173 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.248089    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2587a67-5897-4d57-b72b-7cfcf18b6377-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-rw5l6\" (UID: \"b2587a67-5897-4d57-b72b-7cfcf18b6377\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"
Sep 06 19:29:00.248173 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.248130    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2587a67-5897-4d57-b72b-7cfcf18b6377-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-rw5l6\" (UID: \"b2587a67-5897-4d57-b72b-7cfcf18b6377\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"
Sep 06 19:29:00.248173 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.248156    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8r5k\" (UniqueName: \"kubernetes.io/projected/b2587a67-5897-4d57-b72b-7cfcf18b6377-kube-api-access-t8r5k\") pod \"prometheus-operator-6f4946db56-rw5l6\" (UID: \"b2587a67-5897-4d57-b72b-7cfcf18b6377\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"
Sep 06 19:29:00.248754 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.248183    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2587a67-5897-4d57-b72b-7cfcf18b6377-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-rw5l6\" (UID: \"b2587a67-5897-4d57-b72b-7cfcf18b6377\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"
Sep 06 19:29:00.248754 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:29:00.248266    2571 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 06 19:29:00.248754 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:29:00.248342    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2587a67-5897-4d57-b72b-7cfcf18b6377-prometheus-operator-tls podName:b2587a67-5897-4d57-b72b-7cfcf18b6377 nodeName:}" failed. No retries permitted until 2026-09-06 19:29:00.748320269 +0000 UTC m=+104.318618613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/b2587a67-5897-4d57-b72b-7cfcf18b6377-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-rw5l6" (UID: "b2587a67-5897-4d57-b72b-7cfcf18b6377") : secret "prometheus-operator-tls" not found
Sep 06 19:29:00.248911 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.248847    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2587a67-5897-4d57-b72b-7cfcf18b6377-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-rw5l6\" (UID: \"b2587a67-5897-4d57-b72b-7cfcf18b6377\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"
Sep 06 19:29:00.251143 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.251123    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2587a67-5897-4d57-b72b-7cfcf18b6377-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-rw5l6\" (UID: \"b2587a67-5897-4d57-b72b-7cfcf18b6377\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"
Sep 06 19:29:00.256864 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.256828    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8r5k\" (UniqueName: \"kubernetes.io/projected/b2587a67-5897-4d57-b72b-7cfcf18b6377-kube-api-access-t8r5k\") pod \"prometheus-operator-6f4946db56-rw5l6\" (UID: \"b2587a67-5897-4d57-b72b-7cfcf18b6377\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"
Sep 06 19:29:00.751804 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.751766    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2587a67-5897-4d57-b72b-7cfcf18b6377-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-rw5l6\" (UID: \"b2587a67-5897-4d57-b72b-7cfcf18b6377\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"
Sep 06 19:29:00.754814 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.754771    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2587a67-5897-4d57-b72b-7cfcf18b6377-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-rw5l6\" (UID: \"b2587a67-5897-4d57-b72b-7cfcf18b6377\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"
Sep 06 19:29:00.924382 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:00.924349    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"
Sep 06 19:29:01.236190 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:01.236153    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-rw5l6"]
Sep 06 19:29:01.239887 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:29:01.239860    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2587a67_5897_4d57_b72b_7cfcf18b6377.slice/crio-345192c7747a3748ee258b05eacee8c545f932a584254d9396950a50b2d64c53 WatchSource:0}: Error finding container 345192c7747a3748ee258b05eacee8c545f932a584254d9396950a50b2d64c53: Status 404 returned error can't find the container with id 345192c7747a3748ee258b05eacee8c545f932a584254d9396950a50b2d64c53
Sep 06 19:29:01.379343 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:01.379302    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6" event={"ID":"b2587a67-5897-4d57-b72b-7cfcf18b6377","Type":"ContainerStarted","Data":"345192c7747a3748ee258b05eacee8c545f932a584254d9396950a50b2d64c53"}
Sep 06 19:29:07.366890 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:07.366848    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nfvjj"
Sep 06 19:29:08.406811 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:08.406770    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-wvkbc" event={"ID":"9557b720-35be-4861-8d3a-c82a3acf997e","Type":"ContainerStarted","Data":"50ba0e43a0fb368cfd44070436b81a8ce453b9446878babeb258d0061f9eb455"}
Sep 06 19:29:08.407290 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:08.407136    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-wvkbc"
Sep 06 19:29:08.408549 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:08.408523    2571 patch_prober.go:28] interesting pod/downloads-85d98c998f-wvkbc container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.14:8080/\": dial tcp 10.132.0.14:8080: connect: connection refused" start-of-body=
Sep 06 19:29:08.408675 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:08.408574    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-wvkbc" podUID="9557b720-35be-4861-8d3a-c82a3acf997e" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.14:8080/\": dial tcp 10.132.0.14:8080: connect: connection refused"
Sep 06 19:29:09.415132 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:09.415020    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6" event={"ID":"b2587a67-5897-4d57-b72b-7cfcf18b6377","Type":"ContainerStarted","Data":"fd6562a96372c60253eb935a9ebb8a3c5baebdce82d14d583552fa179f8ad52c"}
Sep 06 19:29:09.415132 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:09.415090    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6" event={"ID":"b2587a67-5897-4d57-b72b-7cfcf18b6377","Type":"ContainerStarted","Data":"0f20ac432f4a8dd83913788e6bfd3c1144ff37b6d10c9592dded564512e50176"}
Sep 06 19:29:09.427495 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:09.427437    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-wvkbc"
Sep 06 19:29:09.432453 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:09.432390    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-rw5l6" podStartSLOduration=2.422421845 podStartE2EDuration="10.432366216s" totalImagesPullingTime="8.009944371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:28:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:29:09.430992133 +0000 UTC m=+113.001290480" watchObservedRunningTime="2026-09-06 19:29:09.432366216 +0000 UTC m=+113.002664565"
Sep 06 19:29:09.432812 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:09.432774    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-wvkbc" podStartSLOduration=1.957621987 podStartE2EDuration="18.432760758s" totalImagesPullingTime="16.475138771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:29:08.428169705 +0000 UTC m=+111.998468052" watchObservedRunningTime="2026-09-06 19:29:09.432760758 +0000 UTC m=+113.003059104"
Sep 06 19:29:10.223279 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:10.223238    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"]
Sep 06 19:29:10.228827 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:10.228798    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:29:11.380771 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.380729    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z46nb"]
Sep 06 19:29:11.412475 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.412443    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.415102 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.415069    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 06 19:29:11.415348 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.415326    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 06 19:29:11.415600 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.415582    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8lnxb\""
Sep 06 19:29:11.415827 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.415808    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 06 19:29:11.541208 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.541167    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-tls\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.541208 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.541224    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2t5w\" (UniqueName: \"kubernetes.io/projected/2864b130-b558-40c7-a2c7-631d0cd7f342-kube-api-access-n2t5w\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.541516 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.541258    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2864b130-b558-40c7-a2c7-631d0cd7f342-sys\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.541516 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.541285    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2864b130-b558-40c7-a2c7-631d0cd7f342-metrics-client-ca\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.541516 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.541319    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.541516 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.541379    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2864b130-b558-40c7-a2c7-631d0cd7f342-root\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.541516 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.541410    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-accelerators-collector-config\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.541516 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.541437    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-textfile\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.541516 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.541467    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-wtmp\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.642309 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.642207    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2864b130-b558-40c7-a2c7-631d0cd7f342-root\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.642309 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.642266    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-accelerators-collector-config\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.642548 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.642354    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2864b130-b558-40c7-a2c7-631d0cd7f342-root\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.642548 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.642394    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-textfile\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.642548 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.642432    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-wtmp\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.642548 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.642467    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-tls\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.642548 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.642498    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2t5w\" (UniqueName: \"kubernetes.io/projected/2864b130-b558-40c7-a2c7-631d0cd7f342-kube-api-access-n2t5w\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.642548 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.642531    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2864b130-b558-40c7-a2c7-631d0cd7f342-sys\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.642837 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.642552    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2864b130-b558-40c7-a2c7-631d0cd7f342-metrics-client-ca\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.642837 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.642585    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.642837 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.642610    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-wtmp\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.642837 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:29:11.642636    2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 06 19:29:11.642837 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.642676    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2864b130-b558-40c7-a2c7-631d0cd7f342-sys\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.642837 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:29:11.642719    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-tls podName:2864b130-b558-40c7-a2c7-631d0cd7f342 nodeName:}" failed. No retries permitted until 2026-09-06 19:29:12.14269664 +0000 UTC m=+115.712994977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-tls") pod "node-exporter-z46nb" (UID: "2864b130-b558-40c7-a2c7-631d0cd7f342") : secret "node-exporter-tls" not found
Sep 06 19:29:11.642837 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.642772    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-textfile\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.643366 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.643338    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2864b130-b558-40c7-a2c7-631d0cd7f342-metrics-client-ca\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.643501 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.643474    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-accelerators-collector-config\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.645985 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.645942    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:11.650743 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:11.650711    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2t5w\" (UniqueName: \"kubernetes.io/projected/2864b130-b558-40c7-a2c7-631d0cd7f342-kube-api-access-n2t5w\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:12.146621 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.146579    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-tls\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:12.149463 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.149431    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2864b130-b558-40c7-a2c7-631d0cd7f342-node-exporter-tls\") pod \"node-exporter-z46nb\" (UID: \"2864b130-b558-40c7-a2c7-631d0cd7f342\") " pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:12.325343 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.325302    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z46nb"
Sep 06 19:29:12.363030 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:29:12.362953    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2864b130_b558_40c7_a2c7_631d0cd7f342.slice/crio-0583a7b27218229e3df0336ed6339e8bc495c5efc5d39e25192b6599b45fe754 WatchSource:0}: Error finding container 0583a7b27218229e3df0336ed6339e8bc495c5efc5d39e25192b6599b45fe754: Status 404 returned error can't find the container with id 0583a7b27218229e3df0336ed6339e8bc495c5efc5d39e25192b6599b45fe754
Sep 06 19:29:12.401197 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.401116    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:29:12.427867 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.427831    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:29:12.428057 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.428004    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z46nb" event={"ID":"2864b130-b558-40c7-a2c7-631d0cd7f342","Type":"ContainerStarted","Data":"0583a7b27218229e3df0336ed6339e8bc495c5efc5d39e25192b6599b45fe754"}
Sep 06 19:29:12.428057 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.428021    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.430464 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.430400    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 06 19:29:12.430648 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.430625    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-867lm\""
Sep 06 19:29:12.430767 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.430696    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 06 19:29:12.431104 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.431083    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 06 19:29:12.431214 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.431156    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 06 19:29:12.431287 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.431217    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 06 19:29:12.431287 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.431163    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 06 19:29:12.431609 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.431567    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 06 19:29:12.431774 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.431759    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 06 19:29:12.432005 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.431989    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 06 19:29:12.550418 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.550385    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.550418 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.550433    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f132da2-88e4-4c7c-8a32-5a72c700b981-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.550662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.550498    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.550662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.550539    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-web-config\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.550662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.550557    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcxpx\" (UniqueName: \"kubernetes.io/projected/9f132da2-88e4-4c7c-8a32-5a72c700b981-kube-api-access-kcxpx\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.550662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.550586    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f132da2-88e4-4c7c-8a32-5a72c700b981-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.550662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.550620    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.550923 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.550716    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-config-volume\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.550923 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.550758    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f132da2-88e4-4c7c-8a32-5a72c700b981-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.550923 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.550807    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.550923 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.550872    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.551140 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.550925    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f132da2-88e4-4c7c-8a32-5a72c700b981-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.551140 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.551008    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f132da2-88e4-4c7c-8a32-5a72c700b981-config-out\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.651983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.651862    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.651983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.651912    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-web-config\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.651983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.651938    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcxpx\" (UniqueName: \"kubernetes.io/projected/9f132da2-88e4-4c7c-8a32-5a72c700b981-kube-api-access-kcxpx\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.651983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.651980    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f132da2-88e4-4c7c-8a32-5a72c700b981-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.652306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.652022    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.652306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.652065    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-config-volume\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.652306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.652096    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f132da2-88e4-4c7c-8a32-5a72c700b981-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.652306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.652128    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.652306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.652170    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.652306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.652216    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f132da2-88e4-4c7c-8a32-5a72c700b981-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.652306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.652245    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f132da2-88e4-4c7c-8a32-5a72c700b981-config-out\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.652306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.652280    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.652764 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.652321    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f132da2-88e4-4c7c-8a32-5a72c700b981-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.654444 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.653831    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f132da2-88e4-4c7c-8a32-5a72c700b981-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.654444 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.654134    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f132da2-88e4-4c7c-8a32-5a72c700b981-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.655121 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.655097    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f132da2-88e4-4c7c-8a32-5a72c700b981-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.655272 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.655209    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-web-config\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.655634 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.655608    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.656194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.656165    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.656610 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.656588    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.657928 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.657518    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.657928 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.657876    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.657928 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.657954    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-config-volume\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.658315 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.658287    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f132da2-88e4-4c7c-8a32-5a72c700b981-config-out\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.659054 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.659033    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f132da2-88e4-4c7c-8a32-5a72c700b981-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.661797 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.661772    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcxpx\" (UniqueName: \"kubernetes.io/projected/9f132da2-88e4-4c7c-8a32-5a72c700b981-kube-api-access-kcxpx\") pod \"alertmanager-main-0\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:12.740585 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:12.740544    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:29:13.066986 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:13.066668    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:29:13.070158 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:29:13.069943    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f132da2_88e4_4c7c_8a32_5a72c700b981.slice/crio-419b624382dfc8007523be7e9540b9a048eb6d11274f519209f4626eb45cfdfb WatchSource:0}: Error finding container 419b624382dfc8007523be7e9540b9a048eb6d11274f519209f4626eb45cfdfb: Status 404 returned error can't find the container with id 419b624382dfc8007523be7e9540b9a048eb6d11274f519209f4626eb45cfdfb
Sep 06 19:29:13.434765 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:13.434670    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f132da2-88e4-4c7c-8a32-5a72c700b981","Type":"ContainerStarted","Data":"419b624382dfc8007523be7e9540b9a048eb6d11274f519209f4626eb45cfdfb"}
Sep 06 19:29:14.443066 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:14.442018    2571 generic.go:356] "Generic (PLEG): container finished" podID="2864b130-b558-40c7-a2c7-631d0cd7f342" containerID="c4315878c86689e70e436c56622b788251924001e0ca87aacb60c76f627c963f" exitCode=0
Sep 06 19:29:14.443066 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:14.442098    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z46nb" event={"ID":"2864b130-b558-40c7-a2c7-631d0cd7f342","Type":"ContainerDied","Data":"c4315878c86689e70e436c56622b788251924001e0ca87aacb60c76f627c963f"}
Sep 06 19:29:15.449431 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.449393    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z46nb" event={"ID":"2864b130-b558-40c7-a2c7-631d0cd7f342","Type":"ContainerStarted","Data":"3297e04543d26ae212d977dc2d091c516b055289ffd405426973eabe8283db5a"}
Sep 06 19:29:15.449910 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.449460    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z46nb" event={"ID":"2864b130-b558-40c7-a2c7-631d0cd7f342","Type":"ContainerStarted","Data":"dc3d1ecbab4d5dc687584f7a190413798d9dade147283bed11c98e8546c974f7"}
Sep 06 19:29:15.451290 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.451261    2571 generic.go:356] "Generic (PLEG): container finished" podID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerID="8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4" exitCode=0
Sep 06 19:29:15.451420 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.451333    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f132da2-88e4-4c7c-8a32-5a72c700b981","Type":"ContainerDied","Data":"8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4"}
Sep 06 19:29:15.467415 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.467364    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z46nb" podStartSLOduration=2.800070078 podStartE2EDuration="4.467342749s" totalImagesPullingTime="1.57892764s" totalInitContainerRuntime="88.345031ms" isStatefulPod=true podCreationTimestamp="2026-09-06 19:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:29:15.466093149 +0000 UTC m=+119.036391495" watchObservedRunningTime="2026-09-06 19:29:15.467342749 +0000 UTC m=+119.037641096"
Sep 06 19:29:15.625194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.625156    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6d889b7db9-gtpf5"]
Sep 06 19:29:15.656027 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.655990    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d889b7db9-gtpf5"]
Sep 06 19:29:15.656202 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.656082    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.658874 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.658845    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 06 19:29:15.659024 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.658896    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-essjl3g1ej02m\""
Sep 06 19:29:15.659024 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.658845    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 06 19:29:15.659024 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.658845    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 06 19:29:15.659206 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.658898    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-74wjl\""
Sep 06 19:29:15.659206 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.658846    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 06 19:29:15.782800 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.782704    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/74fda403-41c5-48fe-92a1-32b1c297c314-secret-metrics-server-tls\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.782800 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.782758    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/74fda403-41c5-48fe-92a1-32b1c297c314-audit-log\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.783055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.782848    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/74fda403-41c5-48fe-92a1-32b1c297c314-secret-metrics-server-client-certs\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.783055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.782904    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74fda403-41c5-48fe-92a1-32b1c297c314-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.783055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.782930    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/74fda403-41c5-48fe-92a1-32b1c297c314-metrics-server-audit-profiles\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.783055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.782954    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncqxf\" (UniqueName: \"kubernetes.io/projected/74fda403-41c5-48fe-92a1-32b1c297c314-kube-api-access-ncqxf\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.783055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.783005    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/74fda403-41c5-48fe-92a1-32b1c297c314-client-ca-bundle\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.884021 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.883973    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/74fda403-41c5-48fe-92a1-32b1c297c314-secret-metrics-server-tls\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.884194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.884041    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/74fda403-41c5-48fe-92a1-32b1c297c314-audit-log\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.884194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.884091    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/74fda403-41c5-48fe-92a1-32b1c297c314-secret-metrics-server-client-certs\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.884194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.884134    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74fda403-41c5-48fe-92a1-32b1c297c314-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.884194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.884164    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/74fda403-41c5-48fe-92a1-32b1c297c314-metrics-server-audit-profiles\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.884194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.884190    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncqxf\" (UniqueName: \"kubernetes.io/projected/74fda403-41c5-48fe-92a1-32b1c297c314-kube-api-access-ncqxf\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.884468 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.884217    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/74fda403-41c5-48fe-92a1-32b1c297c314-client-ca-bundle\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.884800 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.884772    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/74fda403-41c5-48fe-92a1-32b1c297c314-audit-log\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.885205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.885101    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74fda403-41c5-48fe-92a1-32b1c297c314-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.887986 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.886113    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/74fda403-41c5-48fe-92a1-32b1c297c314-metrics-server-audit-profiles\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.887986 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.887386    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/74fda403-41c5-48fe-92a1-32b1c297c314-client-ca-bundle\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.887986 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.887521    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/74fda403-41c5-48fe-92a1-32b1c297c314-secret-metrics-server-tls\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.893172 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.893137    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/74fda403-41c5-48fe-92a1-32b1c297c314-secret-metrics-server-client-certs\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.894157 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.894137    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncqxf\" (UniqueName: \"kubernetes.io/projected/74fda403-41c5-48fe-92a1-32b1c297c314-kube-api-access-ncqxf\") pod \"metrics-server-6d889b7db9-gtpf5\" (UID: \"74fda403-41c5-48fe-92a1-32b1c297c314\") " pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:15.967105 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:15.967069    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:16.297223 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:16.296454    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d889b7db9-gtpf5"]
Sep 06 19:29:16.459118 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:16.458702    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5" event={"ID":"74fda403-41c5-48fe-92a1-32b1c297c314","Type":"ContainerStarted","Data":"c2a138813e13818962e3d46b5430aa99e0ce55fc0cf7794e4fcac05ca1b85e1e"}
Sep 06 19:29:17.502473 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.502435    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 19:29:17.528618 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.528579    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 19:29:17.528825 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.528808    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.534707 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.534679    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 06 19:29:17.535255 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.534684    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 06 19:29:17.535375 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.534718    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7m11301d8hlnm\""
Sep 06 19:29:17.535444 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.534736    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 06 19:29:17.535502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.534775    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 06 19:29:17.535557 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.534811    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 06 19:29:17.535610 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.534829    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 06 19:29:17.535684 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.534835    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 06 19:29:17.536952 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.536579    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 06 19:29:17.536952 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.536694    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 06 19:29:17.536952 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.536731    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 06 19:29:17.536952 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.536807    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4wd79\""
Sep 06 19:29:17.536952 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.536896    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 06 19:29:17.537502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.537275    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 06 19:29:17.537502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.537340    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 06 19:29:17.538721 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.538696    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 06 19:29:17.605589 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.605552    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.605589 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.605594    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.605824 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.605620    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.605824 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.605671    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.605824 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.605722    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3951d4a6-9597-4950-a540-5eee05023666-config-out\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.605824 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.605757    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-web-config\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.606054 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.605858    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.606054 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.605901    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.606054 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.605930    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.606054 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.606004    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-config\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.606054 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.606035    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.606054 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.606054    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.606380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.606069    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.606380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.606092    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.606380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.606113    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.606380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.606149    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3951d4a6-9597-4950-a540-5eee05023666-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.606380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.606190    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cspzz\" (UniqueName: \"kubernetes.io/projected/3951d4a6-9597-4950-a540-5eee05023666-kube-api-access-cspzz\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.606380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.606228    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.606380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.606251    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.606380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.606272    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707296    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3951d4a6-9597-4950-a540-5eee05023666-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707362    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cspzz\" (UniqueName: \"kubernetes.io/projected/3951d4a6-9597-4950-a540-5eee05023666-kube-api-access-cspzz\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707396    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707418    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707442    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707476    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707498    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707524    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707549    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707574    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3951d4a6-9597-4950-a540-5eee05023666-config-out\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707603    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-web-config\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707647    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707675    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707775    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707832    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-config\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707876    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.708243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707906    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.709929 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707929    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.709929 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.707980    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.709929 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.708003    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.710123 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.709973    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.711845 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.710837    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.711845 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.711464    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.712418 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.712397    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.714110 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.713134    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.714110 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.713592    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.714295 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.714121    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.714778 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.714735    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.716595 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.715694    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.716595 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.716184    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.716595 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.716337    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-web-config\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.716595 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.716548    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.716595 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.716557    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.717676 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.717626    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3951d4a6-9597-4950-a540-5eee05023666-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.718158 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.718040    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cspzz\" (UniqueName: \"kubernetes.io/projected/3951d4a6-9597-4950-a540-5eee05023666-kube-api-access-cspzz\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.718335 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.718250    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.718789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.718733    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-config\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.719469 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.719417    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.720062 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.720007    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3951d4a6-9597-4950-a540-5eee05023666-config-out\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.722755 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.722722    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:17.857478 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:17.857442    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:18.177586 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:18.177455    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 19:29:18.470912 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:18.470880    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f132da2-88e4-4c7c-8a32-5a72c700b981","Type":"ContainerStarted","Data":"7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da"}
Sep 06 19:29:18.471061 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:18.470947    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f132da2-88e4-4c7c-8a32-5a72c700b981","Type":"ContainerStarted","Data":"80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0"}
Sep 06 19:29:18.471061 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:18.471012    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f132da2-88e4-4c7c-8a32-5a72c700b981","Type":"ContainerStarted","Data":"a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21"}
Sep 06 19:29:18.471061 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:18.471039    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f132da2-88e4-4c7c-8a32-5a72c700b981","Type":"ContainerStarted","Data":"b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c"}
Sep 06 19:29:18.546213 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:29:18.546181    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3951d4a6_9597_4950_a540_5eee05023666.slice/crio-8f0b8db2bff55df5ca4afba37e3399b99bd874d8fc9a7079b5d8592f18a55f27 WatchSource:0}: Error finding container 8f0b8db2bff55df5ca4afba37e3399b99bd874d8fc9a7079b5d8592f18a55f27: Status 404 returned error can't find the container with id 8f0b8db2bff55df5ca4afba37e3399b99bd874d8fc9a7079b5d8592f18a55f27
Sep 06 19:29:19.477024 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:19.476989    2571 generic.go:356] "Generic (PLEG): container finished" podID="3951d4a6-9597-4950-a540-5eee05023666" containerID="00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1" exitCode=0
Sep 06 19:29:19.477177 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:19.477073    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3951d4a6-9597-4950-a540-5eee05023666","Type":"ContainerDied","Data":"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1"}
Sep 06 19:29:19.477177 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:19.477167    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3951d4a6-9597-4950-a540-5eee05023666","Type":"ContainerStarted","Data":"8f0b8db2bff55df5ca4afba37e3399b99bd874d8fc9a7079b5d8592f18a55f27"}
Sep 06 19:29:19.480862 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:19.480835    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f132da2-88e4-4c7c-8a32-5a72c700b981","Type":"ContainerStarted","Data":"7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725"}
Sep 06 19:29:19.482615 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:19.482590    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5" event={"ID":"74fda403-41c5-48fe-92a1-32b1c297c314","Type":"ContainerStarted","Data":"d75f2f5d796ed077b2bd001aa5d993aebbe7168c3cf2788d3967199073464214"}
Sep 06 19:29:19.517720 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:19.517649    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5" podStartSLOduration=1.9161482429999999 podStartE2EDuration="4.517624429s" totalImagesPullingTime="2.601476186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:29:19.515030643 +0000 UTC m=+123.085328989" watchObservedRunningTime="2026-09-06 19:29:19.517624429 +0000 UTC m=+123.087922775"
Sep 06 19:29:20.492464 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:20.492422    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f132da2-88e4-4c7c-8a32-5a72c700b981","Type":"ContainerStarted","Data":"d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3"}
Sep 06 19:29:20.523833 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:20.523768    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.428961995 podStartE2EDuration="8.523742344s" totalImagesPullingTime="5.072831434s" totalInitContainerRuntime="21.948915ms" isStatefulPod=false podCreationTimestamp="2026-09-06 19:29:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:29:20.520871808 +0000 UTC m=+124.091170155" watchObservedRunningTime="2026-09-06 19:29:20.523742344 +0000 UTC m=+124.094040691"
Sep 06 19:29:23.512076 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:23.512034    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3951d4a6-9597-4950-a540-5eee05023666","Type":"ContainerStarted","Data":"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"}
Sep 06 19:29:23.512477 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:23.512102    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3951d4a6-9597-4950-a540-5eee05023666","Type":"ContainerStarted","Data":"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"}
Sep 06 19:29:26.528575 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:26.528541    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3951d4a6-9597-4950-a540-5eee05023666","Type":"ContainerStarted","Data":"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"}
Sep 06 19:29:26.528947 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:26.528607    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3951d4a6-9597-4950-a540-5eee05023666","Type":"ContainerStarted","Data":"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"}
Sep 06 19:29:26.528947 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:26.528635    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3951d4a6-9597-4950-a540-5eee05023666","Type":"ContainerStarted","Data":"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"}
Sep 06 19:29:26.528947 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:26.528658    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3951d4a6-9597-4950-a540-5eee05023666","Type":"ContainerStarted","Data":"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"}
Sep 06 19:29:26.554184 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:26.554119    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.742673815 podStartE2EDuration="9.554095673s" totalImagesPullingTime="5.790392682s" totalInitContainerRuntime="21.029176ms" isStatefulPod=false podCreationTimestamp="2026-09-06 19:29:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:29:26.552293066 +0000 UTC m=+130.122591411" watchObservedRunningTime="2026-09-06 19:29:26.554095673 +0000 UTC m=+130.124394021"
Sep 06 19:29:26.698746 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:26.698701    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs\") pod \"network-metrics-daemon-pghst\" (UID: \"abf4350f-c25e-43ae-9271-5987625590f7\") " pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:29:26.701129 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:26.701109    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abf4350f-c25e-43ae-9271-5987625590f7-metrics-certs\") pod \"network-metrics-daemon-pghst\" (UID: \"abf4350f-c25e-43ae-9271-5987625590f7\") " pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:29:26.926066 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:26.926029    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9bdz9\""
Sep 06 19:29:26.933871 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:26.933845    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pghst"
Sep 06 19:29:27.245159 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:27.245129    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pghst"]
Sep 06 19:29:27.247392 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:29:27.247366    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf4350f_c25e_43ae_9271_5987625590f7.slice/crio-6d8742140089748375d336be77ee21cc8a03f65a9ba6c1170de029883eda1a4b WatchSource:0}: Error finding container 6d8742140089748375d336be77ee21cc8a03f65a9ba6c1170de029883eda1a4b: Status 404 returned error can't find the container with id 6d8742140089748375d336be77ee21cc8a03f65a9ba6c1170de029883eda1a4b
Sep 06 19:29:27.533829 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:27.533745    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pghst" event={"ID":"abf4350f-c25e-43ae-9271-5987625590f7","Type":"ContainerStarted","Data":"6d8742140089748375d336be77ee21cc8a03f65a9ba6c1170de029883eda1a4b"}
Sep 06 19:29:27.857782 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:27.857750    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:29:29.543797 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:29.543762    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pghst" event={"ID":"abf4350f-c25e-43ae-9271-5987625590f7","Type":"ContainerStarted","Data":"fbf1c60f0812edfa8d9f8278d4c8fe4315a63f411b28d8a12dddd6e3b4cf605a"}
Sep 06 19:29:29.544232 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:29.543818    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pghst" event={"ID":"abf4350f-c25e-43ae-9271-5987625590f7","Type":"ContainerStarted","Data":"8bdbac5645eb9987823e69325831e9f582b7ddebbd8fe46b47181a88615f072c"}
Sep 06 19:29:29.558854 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:29.558801    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pghst" podStartSLOduration=131.203564491 podStartE2EDuration="2m12.558783674s" totalImagesPullingTime="1.355219183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:29:29.557016732 +0000 UTC m=+133.127315078" watchObservedRunningTime="2026-09-06 19:29:29.558783674 +0000 UTC m=+133.129082019"
Sep 06 19:29:35.247832 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.247787    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8" podUID="c379232e-b6a5-4ba3-b941-1232bd22c1f3" containerName="registry" containerID="cri-o://755a0047014d0b894afec9485e684b0aaa6fbca9b47e6876a1d03b110bf5333b" gracePeriod=30
Sep 06 19:29:35.520151 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.520062    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:29:35.568126 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.568091    2571 generic.go:356] "Generic (PLEG): container finished" podID="c379232e-b6a5-4ba3-b941-1232bd22c1f3" containerID="755a0047014d0b894afec9485e684b0aaa6fbca9b47e6876a1d03b110bf5333b" exitCode=0
Sep 06 19:29:35.568311 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.568140    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8" event={"ID":"c379232e-b6a5-4ba3-b941-1232bd22c1f3","Type":"ContainerDied","Data":"755a0047014d0b894afec9485e684b0aaa6fbca9b47e6876a1d03b110bf5333b"}
Sep 06 19:29:35.568311 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.568173    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"
Sep 06 19:29:35.568311 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.568206    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7df7c5cf9c-nd9t8" event={"ID":"c379232e-b6a5-4ba3-b941-1232bd22c1f3","Type":"ContainerDied","Data":"c10fdee18deecd35d41f7fa0d1c466942bae171b5333751f7c4ed879dbd29c04"}
Sep 06 19:29:35.568311 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.568227    2571 scope.go:117] "RemoveContainer" containerID="755a0047014d0b894afec9485e684b0aaa6fbca9b47e6876a1d03b110bf5333b"
Sep 06 19:29:35.576300 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.576266    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls\") pod \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") "
Sep 06 19:29:35.576460 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.576324    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c379232e-b6a5-4ba3-b941-1232bd22c1f3-ca-trust-extracted\") pod \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") "
Sep 06 19:29:35.576460 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.576368    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c379232e-b6a5-4ba3-b941-1232bd22c1f3-trusted-ca\") pod \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") "
Sep 06 19:29:35.576460 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.576403    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvp2m\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-kube-api-access-jvp2m\") pod \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") "
Sep 06 19:29:35.576460 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.576436    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-certificates\") pod \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") "
Sep 06 19:29:35.576676 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.576470    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c379232e-b6a5-4ba3-b941-1232bd22c1f3-installation-pull-secrets\") pod \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") "
Sep 06 19:29:35.576676 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.576524    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-bound-sa-token\") pod \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") "
Sep 06 19:29:35.576676 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.576579    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c379232e-b6a5-4ba3-b941-1232bd22c1f3-image-registry-private-configuration\") pod \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\" (UID: \"c379232e-b6a5-4ba3-b941-1232bd22c1f3\") "
Sep 06 19:29:35.576923 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.576890    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c379232e-b6a5-4ba3-b941-1232bd22c1f3" (UID: "c379232e-b6a5-4ba3-b941-1232bd22c1f3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:29:35.577431 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.577406    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c379232e-b6a5-4ba3-b941-1232bd22c1f3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c379232e-b6a5-4ba3-b941-1232bd22c1f3" (UID: "c379232e-b6a5-4ba3-b941-1232bd22c1f3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:29:35.579100 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.579069    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c379232e-b6a5-4ba3-b941-1232bd22c1f3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c379232e-b6a5-4ba3-b941-1232bd22c1f3" (UID: "c379232e-b6a5-4ba3-b941-1232bd22c1f3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:29:35.579221 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.579157    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c379232e-b6a5-4ba3-b941-1232bd22c1f3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c379232e-b6a5-4ba3-b941-1232bd22c1f3" (UID: "c379232e-b6a5-4ba3-b941-1232bd22c1f3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:29:35.579386 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.579361    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c379232e-b6a5-4ba3-b941-1232bd22c1f3" (UID: "c379232e-b6a5-4ba3-b941-1232bd22c1f3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:29:35.579501 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.579469    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c379232e-b6a5-4ba3-b941-1232bd22c1f3" (UID: "c379232e-b6a5-4ba3-b941-1232bd22c1f3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:29:35.580805 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.580779    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-kube-api-access-jvp2m" (OuterVolumeSpecName: "kube-api-access-jvp2m") pod "c379232e-b6a5-4ba3-b941-1232bd22c1f3" (UID: "c379232e-b6a5-4ba3-b941-1232bd22c1f3"). InnerVolumeSpecName "kube-api-access-jvp2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:29:35.581324 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.581308    2571 scope.go:117] "RemoveContainer" containerID="755a0047014d0b894afec9485e684b0aaa6fbca9b47e6876a1d03b110bf5333b"
Sep 06 19:29:35.581727 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:29:35.581704    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"755a0047014d0b894afec9485e684b0aaa6fbca9b47e6876a1d03b110bf5333b\": container with ID starting with 755a0047014d0b894afec9485e684b0aaa6fbca9b47e6876a1d03b110bf5333b not found: ID does not exist" containerID="755a0047014d0b894afec9485e684b0aaa6fbca9b47e6876a1d03b110bf5333b"
Sep 06 19:29:35.581781 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.581736    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755a0047014d0b894afec9485e684b0aaa6fbca9b47e6876a1d03b110bf5333b"} err="failed to get container status \"755a0047014d0b894afec9485e684b0aaa6fbca9b47e6876a1d03b110bf5333b\": rpc error: code = NotFound desc = could not find container \"755a0047014d0b894afec9485e684b0aaa6fbca9b47e6876a1d03b110bf5333b\": container with ID starting with 755a0047014d0b894afec9485e684b0aaa6fbca9b47e6876a1d03b110bf5333b not found: ID does not exist"
Sep 06 19:29:35.586718 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.586690    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c379232e-b6a5-4ba3-b941-1232bd22c1f3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c379232e-b6a5-4ba3-b941-1232bd22c1f3" (UID: "c379232e-b6a5-4ba3-b941-1232bd22c1f3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:29:35.677778 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.677734    2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c379232e-b6a5-4ba3-b941-1232bd22c1f3-installation-pull-secrets\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:29:35.677778 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.677771    2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-bound-sa-token\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:29:35.677778 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.677784    2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c379232e-b6a5-4ba3-b941-1232bd22c1f3-image-registry-private-configuration\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:29:35.678053 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.677800    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-tls\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:29:35.678053 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.677813    2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c379232e-b6a5-4ba3-b941-1232bd22c1f3-ca-trust-extracted\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:29:35.678053 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.677823    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c379232e-b6a5-4ba3-b941-1232bd22c1f3-trusted-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:29:35.678053 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.677834    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvp2m\" (UniqueName: \"kubernetes.io/projected/c379232e-b6a5-4ba3-b941-1232bd22c1f3-kube-api-access-jvp2m\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:29:35.678053 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.677844    2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c379232e-b6a5-4ba3-b941-1232bd22c1f3-registry-certificates\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:29:35.888470 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.888438    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"]
Sep 06 19:29:35.891577 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.891551    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7df7c5cf9c-nd9t8"]
Sep 06 19:29:35.967689 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.967647    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:35.967876 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:35.967712    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:36.917108 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:36.917070    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c379232e-b6a5-4ba3-b941-1232bd22c1f3" path="/var/lib/kubelet/pods/c379232e-b6a5-4ba3-b941-1232bd22c1f3/volumes"
Sep 06 19:29:55.973461 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:55.973426    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:29:55.977625 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:29:55.977595    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6d889b7db9-gtpf5"
Sep 06 19:30:17.858332 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:17.858290    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:17.878450 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:17.878421    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:18.760946 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:18.760917    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:31.553513 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:31.553475    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:30:31.554232 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:31.553854    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="alertmanager" containerID="cri-o://b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c" gracePeriod=120
Sep 06 19:30:31.554232 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:31.553915    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="kube-rbac-proxy-web" containerID="cri-o://80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0" gracePeriod=120
Sep 06 19:30:31.554232 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:31.553948    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="prom-label-proxy" containerID="cri-o://d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3" gracePeriod=120
Sep 06 19:30:31.554232 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:31.553915    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="kube-rbac-proxy-metric" containerID="cri-o://7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725" gracePeriod=120
Sep 06 19:30:31.554232 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:31.554079    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="config-reloader" containerID="cri-o://a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21" gracePeriod=120
Sep 06 19:30:31.554232 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:31.554165    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="kube-rbac-proxy" containerID="cri-o://7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da" gracePeriod=120
Sep 06 19:30:31.799126 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:31.799093    2571 generic.go:356] "Generic (PLEG): container finished" podID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerID="d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3" exitCode=0
Sep 06 19:30:31.799126 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:31.799122    2571 generic.go:356] "Generic (PLEG): container finished" podID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerID="7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da" exitCode=0
Sep 06 19:30:31.799126 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:31.799129    2571 generic.go:356] "Generic (PLEG): container finished" podID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerID="a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21" exitCode=0
Sep 06 19:30:31.799126 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:31.799135    2571 generic.go:356] "Generic (PLEG): container finished" podID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerID="b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c" exitCode=0
Sep 06 19:30:31.799442 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:31.799168    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f132da2-88e4-4c7c-8a32-5a72c700b981","Type":"ContainerDied","Data":"d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3"}
Sep 06 19:30:31.799442 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:31.799249    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f132da2-88e4-4c7c-8a32-5a72c700b981","Type":"ContainerDied","Data":"7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da"}
Sep 06 19:30:31.799442 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:31.799289    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f132da2-88e4-4c7c-8a32-5a72c700b981","Type":"ContainerDied","Data":"a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21"}
Sep 06 19:30:31.799442 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:31.799326    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f132da2-88e4-4c7c-8a32-5a72c700b981","Type":"ContainerDied","Data":"b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c"}
Sep 06 19:30:32.795285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.795253    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:32.806652 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.806621    2571 generic.go:356] "Generic (PLEG): container finished" podID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerID="7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725" exitCode=0
Sep 06 19:30:32.806652 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.806649    2571 generic.go:356] "Generic (PLEG): container finished" podID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerID="80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0" exitCode=0
Sep 06 19:30:32.806843 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.806657    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f132da2-88e4-4c7c-8a32-5a72c700b981","Type":"ContainerDied","Data":"7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725"}
Sep 06 19:30:32.806843 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.806702    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:32.806843 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.806774    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f132da2-88e4-4c7c-8a32-5a72c700b981","Type":"ContainerDied","Data":"80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0"}
Sep 06 19:30:32.806843 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.806808    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f132da2-88e4-4c7c-8a32-5a72c700b981","Type":"ContainerDied","Data":"419b624382dfc8007523be7e9540b9a048eb6d11274f519209f4626eb45cfdfb"}
Sep 06 19:30:32.806843 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.806838    2571 scope.go:117] "RemoveContainer" containerID="d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3"
Sep 06 19:30:32.814416 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.814393    2571 scope.go:117] "RemoveContainer" containerID="7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725"
Sep 06 19:30:32.822555 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.822489    2571 scope.go:117] "RemoveContainer" containerID="7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da"
Sep 06 19:30:32.831806 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.831781    2571 scope.go:117] "RemoveContainer" containerID="80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0"
Sep 06 19:30:32.840313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.840289    2571 scope.go:117] "RemoveContainer" containerID="a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21"
Sep 06 19:30:32.848088 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.848068    2571 scope.go:117] "RemoveContainer" containerID="b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c"
Sep 06 19:30:32.855378 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.855358    2571 scope.go:117] "RemoveContainer" containerID="8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4"
Sep 06 19:30:32.863326 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.863292    2571 scope.go:117] "RemoveContainer" containerID="d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3"
Sep 06 19:30:32.863629 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:30:32.863609    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3\": container with ID starting with d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3 not found: ID does not exist" containerID="d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3"
Sep 06 19:30:32.863704 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.863643    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3"} err="failed to get container status \"d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3\": rpc error: code = NotFound desc = could not find container \"d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3\": container with ID starting with d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3 not found: ID does not exist"
Sep 06 19:30:32.863704 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.863675    2571 scope.go:117] "RemoveContainer" containerID="7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725"
Sep 06 19:30:32.864061 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:30:32.864040    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725\": container with ID starting with 7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725 not found: ID does not exist" containerID="7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725"
Sep 06 19:30:32.864175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.864070    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725"} err="failed to get container status \"7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725\": rpc error: code = NotFound desc = could not find container \"7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725\": container with ID starting with 7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725 not found: ID does not exist"
Sep 06 19:30:32.864175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.864085    2571 scope.go:117] "RemoveContainer" containerID="7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da"
Sep 06 19:30:32.864384 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:30:32.864362    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da\": container with ID starting with 7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da not found: ID does not exist" containerID="7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da"
Sep 06 19:30:32.864425 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.864395    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da"} err="failed to get container status \"7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da\": rpc error: code = NotFound desc = could not find container \"7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da\": container with ID starting with 7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da not found: ID does not exist"
Sep 06 19:30:32.864425 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.864416    2571 scope.go:117] "RemoveContainer" containerID="80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0"
Sep 06 19:30:32.864732 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:30:32.864710    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0\": container with ID starting with 80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0 not found: ID does not exist" containerID="80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0"
Sep 06 19:30:32.864795 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.864739    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0"} err="failed to get container status \"80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0\": rpc error: code = NotFound desc = could not find container \"80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0\": container with ID starting with 80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0 not found: ID does not exist"
Sep 06 19:30:32.864795 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.864760    2571 scope.go:117] "RemoveContainer" containerID="a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21"
Sep 06 19:30:32.865037 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:30:32.865010    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21\": container with ID starting with a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21 not found: ID does not exist" containerID="a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21"
Sep 06 19:30:32.865131 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.865041    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21"} err="failed to get container status \"a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21\": rpc error: code = NotFound desc = could not find container \"a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21\": container with ID starting with a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21 not found: ID does not exist"
Sep 06 19:30:32.865131 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.865062    2571 scope.go:117] "RemoveContainer" containerID="b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c"
Sep 06 19:30:32.865327 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:30:32.865307    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c\": container with ID starting with b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c not found: ID does not exist" containerID="b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c"
Sep 06 19:30:32.865413 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.865332    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c"} err="failed to get container status \"b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c\": rpc error: code = NotFound desc = could not find container \"b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c\": container with ID starting with b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c not found: ID does not exist"
Sep 06 19:30:32.865413 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.865352    2571 scope.go:117] "RemoveContainer" containerID="8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4"
Sep 06 19:30:32.865649 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:30:32.865633    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4\": container with ID starting with 8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4 not found: ID does not exist" containerID="8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4"
Sep 06 19:30:32.865707 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.865651    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4"} err="failed to get container status \"8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4\": rpc error: code = NotFound desc = could not find container \"8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4\": container with ID starting with 8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4 not found: ID does not exist"
Sep 06 19:30:32.865707 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.865669    2571 scope.go:117] "RemoveContainer" containerID="d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3"
Sep 06 19:30:32.865928 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.865909    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3"} err="failed to get container status \"d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3\": rpc error: code = NotFound desc = could not find container \"d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3\": container with ID starting with d7e691b5a954fa6d8018f5128225539b9e96652933b45dacc712d91e0a43f3a3 not found: ID does not exist"
Sep 06 19:30:32.866039 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.865928    2571 scope.go:117] "RemoveContainer" containerID="7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725"
Sep 06 19:30:32.866348 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.866325    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725"} err="failed to get container status \"7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725\": rpc error: code = NotFound desc = could not find container \"7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725\": container with ID starting with 7a8cf9c4c8f78e3c83a43b9891cc7dd1bc839208a1e1212b72da76d9f2c47725 not found: ID does not exist"
Sep 06 19:30:32.866399 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.866350    2571 scope.go:117] "RemoveContainer" containerID="7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da"
Sep 06 19:30:32.866651 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.866631    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da"} err="failed to get container status \"7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da\": rpc error: code = NotFound desc = could not find container \"7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da\": container with ID starting with 7df532381ed1fed159fc9d7f4edc90fb99a2fcd9ba6e0a5553d0170dfd5839da not found: ID does not exist"
Sep 06 19:30:32.866651 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.866650    2571 scope.go:117] "RemoveContainer" containerID="80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0"
Sep 06 19:30:32.866885 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.866868    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0"} err="failed to get container status \"80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0\": rpc error: code = NotFound desc = could not find container \"80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0\": container with ID starting with 80187926f70daaf1a919315ada31bfe41aba24862fada1f58a71a637e90691f0 not found: ID does not exist"
Sep 06 19:30:32.866945 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.866885    2571 scope.go:117] "RemoveContainer" containerID="a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21"
Sep 06 19:30:32.867151 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.867120    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21"} err="failed to get container status \"a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21\": rpc error: code = NotFound desc = could not find container \"a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21\": container with ID starting with a9ad1d52a658528dde8a643c6983d707a5992cc68f3768f7255d79f192cfea21 not found: ID does not exist"
Sep 06 19:30:32.867151 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.867148    2571 scope.go:117] "RemoveContainer" containerID="b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c"
Sep 06 19:30:32.867474 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.867443    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c"} err="failed to get container status \"b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c\": rpc error: code = NotFound desc = could not find container \"b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c\": container with ID starting with b17cd0868d4c4752352520a843868109233743ce5e5c06e3a49f8f294cc38a7c not found: ID does not exist"
Sep 06 19:30:32.867474 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.867471    2571 scope.go:117] "RemoveContainer" containerID="8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4"
Sep 06 19:30:32.867748 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.867726    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4"} err="failed to get container status \"8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4\": rpc error: code = NotFound desc = could not find container \"8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4\": container with ID starting with 8ea5e3c9721c34ab2f619d6d5754f984983746bb6d020bf04f6c91e4a3e09df4 not found: ID does not exist"
Sep 06 19:30:32.974390 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.974354    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-config-volume\") pod \"9f132da2-88e4-4c7c-8a32-5a72c700b981\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") "
Sep 06 19:30:32.974390 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.974398    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-cluster-tls-config\") pod \"9f132da2-88e4-4c7c-8a32-5a72c700b981\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") "
Sep 06 19:30:32.974600 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.974424    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f132da2-88e4-4c7c-8a32-5a72c700b981-alertmanager-trusted-ca-bundle\") pod \"9f132da2-88e4-4c7c-8a32-5a72c700b981\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") "
Sep 06 19:30:32.974600 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.974458    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f132da2-88e4-4c7c-8a32-5a72c700b981-alertmanager-main-db\") pod \"9f132da2-88e4-4c7c-8a32-5a72c700b981\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") "
Sep 06 19:30:32.974600 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.974576    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy\") pod \"9f132da2-88e4-4c7c-8a32-5a72c700b981\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") "
Sep 06 19:30:32.974745 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.974624    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f132da2-88e4-4c7c-8a32-5a72c700b981-tls-assets\") pod \"9f132da2-88e4-4c7c-8a32-5a72c700b981\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") "
Sep 06 19:30:32.974745 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.974655    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcxpx\" (UniqueName: \"kubernetes.io/projected/9f132da2-88e4-4c7c-8a32-5a72c700b981-kube-api-access-kcxpx\") pod \"9f132da2-88e4-4c7c-8a32-5a72c700b981\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") "
Sep 06 19:30:32.974745 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.974688    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f132da2-88e4-4c7c-8a32-5a72c700b981-metrics-client-ca\") pod \"9f132da2-88e4-4c7c-8a32-5a72c700b981\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") "
Sep 06 19:30:32.974745 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.974714    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9f132da2-88e4-4c7c-8a32-5a72c700b981\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") "
Sep 06 19:30:32.974942 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.974751    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f132da2-88e4-4c7c-8a32-5a72c700b981-config-out\") pod \"9f132da2-88e4-4c7c-8a32-5a72c700b981\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") "
Sep 06 19:30:32.974942 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.974760    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f132da2-88e4-4c7c-8a32-5a72c700b981-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "9f132da2-88e4-4c7c-8a32-5a72c700b981" (UID: "9f132da2-88e4-4c7c-8a32-5a72c700b981"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:30:32.974942 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.974807    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-web-config\") pod \"9f132da2-88e4-4c7c-8a32-5a72c700b981\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") "
Sep 06 19:30:32.974942 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.974842    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy-web\") pod \"9f132da2-88e4-4c7c-8a32-5a72c700b981\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") "
Sep 06 19:30:32.974942 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.974895    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-main-tls\") pod \"9f132da2-88e4-4c7c-8a32-5a72c700b981\" (UID: \"9f132da2-88e4-4c7c-8a32-5a72c700b981\") "
Sep 06 19:30:32.974942 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.974911    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f132da2-88e4-4c7c-8a32-5a72c700b981-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "9f132da2-88e4-4c7c-8a32-5a72c700b981" (UID: "9f132da2-88e4-4c7c-8a32-5a72c700b981"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:30:32.975291 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.975149    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f132da2-88e4-4c7c-8a32-5a72c700b981-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "9f132da2-88e4-4c7c-8a32-5a72c700b981" (UID: "9f132da2-88e4-4c7c-8a32-5a72c700b981"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:30:32.975291 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.975203    2571 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f132da2-88e4-4c7c-8a32-5a72c700b981-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:32.975291 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.975228    2571 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f132da2-88e4-4c7c-8a32-5a72c700b981-alertmanager-main-db\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:32.977355 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.977321    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f132da2-88e4-4c7c-8a32-5a72c700b981-kube-api-access-kcxpx" (OuterVolumeSpecName: "kube-api-access-kcxpx") pod "9f132da2-88e4-4c7c-8a32-5a72c700b981" (UID: "9f132da2-88e4-4c7c-8a32-5a72c700b981"). InnerVolumeSpecName "kube-api-access-kcxpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:30:32.977849 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.977824    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-config-volume" (OuterVolumeSpecName: "config-volume") pod "9f132da2-88e4-4c7c-8a32-5a72c700b981" (UID: "9f132da2-88e4-4c7c-8a32-5a72c700b981"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:32.978074 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.978039    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f132da2-88e4-4c7c-8a32-5a72c700b981-config-out" (OuterVolumeSpecName: "config-out") pod "9f132da2-88e4-4c7c-8a32-5a72c700b981" (UID: "9f132da2-88e4-4c7c-8a32-5a72c700b981"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:30:32.978161 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.978087    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f132da2-88e4-4c7c-8a32-5a72c700b981-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9f132da2-88e4-4c7c-8a32-5a72c700b981" (UID: "9f132da2-88e4-4c7c-8a32-5a72c700b981"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:30:32.978360 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.978340    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "9f132da2-88e4-4c7c-8a32-5a72c700b981" (UID: "9f132da2-88e4-4c7c-8a32-5a72c700b981"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:32.978435 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.978366    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "9f132da2-88e4-4c7c-8a32-5a72c700b981" (UID: "9f132da2-88e4-4c7c-8a32-5a72c700b981"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:32.978893 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.978875    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "9f132da2-88e4-4c7c-8a32-5a72c700b981" (UID: "9f132da2-88e4-4c7c-8a32-5a72c700b981"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:32.979778 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.979756    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "9f132da2-88e4-4c7c-8a32-5a72c700b981" (UID: "9f132da2-88e4-4c7c-8a32-5a72c700b981"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:32.982305 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.982284    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "9f132da2-88e4-4c7c-8a32-5a72c700b981" (UID: "9f132da2-88e4-4c7c-8a32-5a72c700b981"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:32.988814 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:32.988778    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-web-config" (OuterVolumeSpecName: "web-config") pod "9f132da2-88e4-4c7c-8a32-5a72c700b981" (UID: "9f132da2-88e4-4c7c-8a32-5a72c700b981"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:33.076309 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.076259    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:33.076309 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.076305    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-main-tls\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:33.076309 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.076316    2571 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-config-volume\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:33.076522 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.076325    2571 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-cluster-tls-config\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:33.076522 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.076338    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:33.076522 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.076346    2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f132da2-88e4-4c7c-8a32-5a72c700b981-tls-assets\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:33.076522 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.076355    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcxpx\" (UniqueName: \"kubernetes.io/projected/9f132da2-88e4-4c7c-8a32-5a72c700b981-kube-api-access-kcxpx\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:33.076522 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.076363    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f132da2-88e4-4c7c-8a32-5a72c700b981-metrics-client-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:33.076522 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.076372    2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:33.076522 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.076381    2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f132da2-88e4-4c7c-8a32-5a72c700b981-config-out\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:33.076522 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.076391    2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f132da2-88e4-4c7c-8a32-5a72c700b981-web-config\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:33.129260 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.129224    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:30:33.132359 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.132323    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:30:33.154019 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.153985    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:30:33.154261 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154249    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="kube-rbac-proxy-web"
Sep 06 19:30:33.154302 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154263    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="kube-rbac-proxy-web"
Sep 06 19:30:33.154302 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154271    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="prom-label-proxy"
Sep 06 19:30:33.154302 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154277    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="prom-label-proxy"
Sep 06 19:30:33.154422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154317    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="alertmanager"
Sep 06 19:30:33.154422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154324    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="alertmanager"
Sep 06 19:30:33.154422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154332    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c379232e-b6a5-4ba3-b941-1232bd22c1f3" containerName="registry"
Sep 06 19:30:33.154422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154337    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c379232e-b6a5-4ba3-b941-1232bd22c1f3" containerName="registry"
Sep 06 19:30:33.154422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154344    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="init-config-reloader"
Sep 06 19:30:33.154422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154350    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="init-config-reloader"
Sep 06 19:30:33.154422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154355    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="kube-rbac-proxy"
Sep 06 19:30:33.154422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154360    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="kube-rbac-proxy"
Sep 06 19:30:33.154422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154370    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="config-reloader"
Sep 06 19:30:33.154422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154375    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="config-reloader"
Sep 06 19:30:33.154422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154385    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="kube-rbac-proxy-metric"
Sep 06 19:30:33.154422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154391    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="kube-rbac-proxy-metric"
Sep 06 19:30:33.154840 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154468    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="kube-rbac-proxy-web"
Sep 06 19:30:33.154840 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154477    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="config-reloader"
Sep 06 19:30:33.154840 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154483    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="kube-rbac-proxy"
Sep 06 19:30:33.154840 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154488    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="prom-label-proxy"
Sep 06 19:30:33.154840 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154493    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c379232e-b6a5-4ba3-b941-1232bd22c1f3" containerName="registry"
Sep 06 19:30:33.154840 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154500    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="alertmanager"
Sep 06 19:30:33.154840 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.154504    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" containerName="kube-rbac-proxy-metric"
Sep 06 19:30:33.158968 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.158935    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.161420 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.161389    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 06 19:30:33.161545 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.161468    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 06 19:30:33.161771 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.161752    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 06 19:30:33.162167 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.162146    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-867lm\""
Sep 06 19:30:33.162167 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.162155    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 06 19:30:33.162316 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.162193    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 06 19:30:33.162316 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.162150    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 06 19:30:33.162316 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.162193    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 06 19:30:33.162316 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.162244    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 06 19:30:33.171251 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.169345    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 06 19:30:33.174755 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.174731    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:30:33.278539 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.278500    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nx5t\" (UniqueName: \"kubernetes.io/projected/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-kube-api-access-9nx5t\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.278709 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.278560    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.278709 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.278599    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.278709 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.278623    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.278709 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.278646    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.278709 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.278661    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.278897 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.278718    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-config-out\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.278897 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.278782    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-config-volume\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.278897 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.278800    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.278897 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.278822    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.278897 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.278844    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-web-config\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.279106 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.278898    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.279106 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.278925    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.379736 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.379638    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-config-volume\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.379736 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.379683    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.379736 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.379716    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.380053 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.379909    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-web-config\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.380053 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.379996    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.380053 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.380037    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.380198 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.380077    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nx5t\" (UniqueName: \"kubernetes.io/projected/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-kube-api-access-9nx5t\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.380198 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.380121    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.380198 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.380164    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.380343 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.380205    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.380343 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.380243    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.380343 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.380265    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.380343 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.380300    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-config-out\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.380533 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.380382    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.380820 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.380765    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.381460 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.381421    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.382748 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.382627    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-web-config\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.382748 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.382692    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.382922 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.382802    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-config-out\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.382922 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.382874    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.383053 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.383004    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.383191 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.383167    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.383942 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.383922    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.384490 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.384469    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-config-volume\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.384896 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.384878    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.387503 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.387486    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nx5t\" (UniqueName: \"kubernetes.io/projected/91955a6f-f51f-4bfd-8647-776c9c7e2a8b-kube-api-access-9nx5t\") pod \"alertmanager-main-0\" (UID: \"91955a6f-f51f-4bfd-8647-776c9c7e2a8b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.474773 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.474737    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 06 19:30:33.749593 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.749521    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 06 19:30:33.752110 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:30:33.752083    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91955a6f_f51f_4bfd_8647_776c9c7e2a8b.slice/crio-df97a9fb7fef38dafeefc87345cadb0f6f5d1998b0b0905b061c28765f1e9c14 WatchSource:0}: Error finding container df97a9fb7fef38dafeefc87345cadb0f6f5d1998b0b0905b061c28765f1e9c14: Status 404 returned error can't find the container with id df97a9fb7fef38dafeefc87345cadb0f6f5d1998b0b0905b061c28765f1e9c14
Sep 06 19:30:33.814508 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:33.814477    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91955a6f-f51f-4bfd-8647-776c9c7e2a8b","Type":"ContainerStarted","Data":"df97a9fb7fef38dafeefc87345cadb0f6f5d1998b0b0905b061c28765f1e9c14"}
Sep 06 19:30:34.819663 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:34.819632    2571 generic.go:356] "Generic (PLEG): container finished" podID="91955a6f-f51f-4bfd-8647-776c9c7e2a8b" containerID="033bd1d62427a1d2933e642a24e9899aaf7250312670848093cbfe49291c959b" exitCode=0
Sep 06 19:30:34.819663 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:34.819670    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91955a6f-f51f-4bfd-8647-776c9c7e2a8b","Type":"ContainerDied","Data":"033bd1d62427a1d2933e642a24e9899aaf7250312670848093cbfe49291c959b"}
Sep 06 19:30:34.919050 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:34.919019    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f132da2-88e4-4c7c-8a32-5a72c700b981" path="/var/lib/kubelet/pods/9f132da2-88e4-4c7c-8a32-5a72c700b981/volumes"
Sep 06 19:30:35.570663 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.565211    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"]
Sep 06 19:30:35.570663 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.569289    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.572913 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.572879    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 06 19:30:35.573082 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.572988    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 06 19:30:35.573170 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.573156    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 06 19:30:35.573243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.573222    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-dm667\""
Sep 06 19:30:35.573416 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.573396    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 06 19:30:35.573718 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.573696    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 06 19:30:35.577380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.577357    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 06 19:30:35.582888 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.582350    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"]
Sep 06 19:30:35.701745 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.701708    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/81670dcf-0538-45c8-897d-91fd12a87bea-federate-client-tls\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.701745 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.701749    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/81670dcf-0538-45c8-897d-91fd12a87bea-telemeter-client-tls\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.702006 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.701772    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81670dcf-0538-45c8-897d-91fd12a87bea-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.702006 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.701827    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81670dcf-0538-45c8-897d-91fd12a87bea-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.702006 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.701939    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw766\" (UniqueName: \"kubernetes.io/projected/81670dcf-0538-45c8-897d-91fd12a87bea-kube-api-access-mw766\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.702114 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.702011    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81670dcf-0538-45c8-897d-91fd12a87bea-serving-certs-ca-bundle\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.702114 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.702043    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81670dcf-0538-45c8-897d-91fd12a87bea-metrics-client-ca\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.702114 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.702080    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/81670dcf-0538-45c8-897d-91fd12a87bea-secret-telemeter-client\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.803024 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.802985    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/81670dcf-0538-45c8-897d-91fd12a87bea-federate-client-tls\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.803024 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.803026    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/81670dcf-0538-45c8-897d-91fd12a87bea-telemeter-client-tls\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.803273 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.803046    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81670dcf-0538-45c8-897d-91fd12a87bea-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.803273 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.803063    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81670dcf-0538-45c8-897d-91fd12a87bea-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.803273 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.803112    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw766\" (UniqueName: \"kubernetes.io/projected/81670dcf-0538-45c8-897d-91fd12a87bea-kube-api-access-mw766\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.803273 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.803134    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81670dcf-0538-45c8-897d-91fd12a87bea-serving-certs-ca-bundle\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.803273 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.803153    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81670dcf-0538-45c8-897d-91fd12a87bea-metrics-client-ca\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.803273 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.803177    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/81670dcf-0538-45c8-897d-91fd12a87bea-secret-telemeter-client\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.803992 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.803920    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81670dcf-0538-45c8-897d-91fd12a87bea-metrics-client-ca\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.804129 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.804099    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81670dcf-0538-45c8-897d-91fd12a87bea-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.804258 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.804234    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81670dcf-0538-45c8-897d-91fd12a87bea-serving-certs-ca-bundle\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.805938 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.805909    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81670dcf-0538-45c8-897d-91fd12a87bea-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.806101 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.806078    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/81670dcf-0538-45c8-897d-91fd12a87bea-secret-telemeter-client\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.806164 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.806149    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/81670dcf-0538-45c8-897d-91fd12a87bea-telemeter-client-tls\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.806217 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.806166    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/81670dcf-0538-45c8-897d-91fd12a87bea-federate-client-tls\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.809944 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.809922    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw766\" (UniqueName: \"kubernetes.io/projected/81670dcf-0538-45c8-897d-91fd12a87bea-kube-api-access-mw766\") pod \"telemeter-client-6df9f98f87-jgzb5\" (UID: \"81670dcf-0538-45c8-897d-91fd12a87bea\") " pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:35.829625 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.829535    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91955a6f-f51f-4bfd-8647-776c9c7e2a8b","Type":"ContainerStarted","Data":"4280c8e846c191197ac8f525d823094974f24f2b5e4c23dcd95f84138bee0b49"}
Sep 06 19:30:35.829625 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.829607    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91955a6f-f51f-4bfd-8647-776c9c7e2a8b","Type":"ContainerStarted","Data":"e283125013d864c4f61c6b18e75493d175acb4bb10c329bb16358e8cacbf693c"}
Sep 06 19:30:35.830157 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.829633    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91955a6f-f51f-4bfd-8647-776c9c7e2a8b","Type":"ContainerStarted","Data":"07829677a00347232ec706ef2d705d36af10be34cc3d77ac9d370356f0c46d13"}
Sep 06 19:30:35.830157 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.829658    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91955a6f-f51f-4bfd-8647-776c9c7e2a8b","Type":"ContainerStarted","Data":"0e3e1b3eab9189982c51af83815b746b1d4855622b44346530b69beb669791ee"}
Sep 06 19:30:35.830157 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.829688    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91955a6f-f51f-4bfd-8647-776c9c7e2a8b","Type":"ContainerStarted","Data":"84a29639efcecc5c58d9955a82b6c4d9561a93401a94edd5541f02d16ddc46d1"}
Sep 06 19:30:35.830157 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.829713    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"91955a6f-f51f-4bfd-8647-776c9c7e2a8b","Type":"ContainerStarted","Data":"ec4e739f6b05a526d84563ce7b823b8908fa71147b0c4eccdfdde45d1188eb8b"}
Sep 06 19:30:35.845304 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.845266    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 19:30:35.845723 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.845684    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="thanos-sidecar" containerID="cri-o://4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a" gracePeriod=600
Sep 06 19:30:35.845887 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.845747    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="kube-rbac-proxy-thanos" containerID="cri-o://bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c" gracePeriod=600
Sep 06 19:30:35.845887 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.845699    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="kube-rbac-proxy" containerID="cri-o://52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961" gracePeriod=600
Sep 06 19:30:35.845887 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.845793    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="prometheus" containerID="cri-o://99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4" gracePeriod=600
Sep 06 19:30:35.845887 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.845686    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="config-reloader" containerID="cri-o://a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2" gracePeriod=600
Sep 06 19:30:35.845887 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.845747    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="kube-rbac-proxy-web" containerID="cri-o://a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106" gracePeriod=600
Sep 06 19:30:35.859379 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.858062    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.844137762 podStartE2EDuration="2.858040125s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.902363ms" isStatefulPod=false podCreationTimestamp="2026-09-06 19:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:30:35.856294301 +0000 UTC m=+199.426592646" watchObservedRunningTime="2026-09-06 19:30:35.858040125 +0000 UTC m=+199.428338471"
Sep 06 19:30:35.882268 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:35.882019    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"
Sep 06 19:30:36.104234 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.104206    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.175093 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.175067    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6df9f98f87-jgzb5"]
Sep 06 19:30:36.177458 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:30:36.177415    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81670dcf_0538_45c8_897d_91fd12a87bea.slice/crio-b9d723ee8a1e55888b126d0e6d8e5331a4c403c3fad1d19bd3a13750e65e9832 WatchSource:0}: Error finding container b9d723ee8a1e55888b126d0e6d8e5331a4c403c3fad1d19bd3a13750e65e9832: Status 404 returned error can't find the container with id b9d723ee8a1e55888b126d0e6d8e5331a4c403c3fad1d19bd3a13750e65e9832
Sep 06 19:30:36.208392 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.208359    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-web-config\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.208392 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.208397    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-1\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.208640 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.208418    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-0\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.208640 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.208442    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-tls\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.208640 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.208461    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3951d4a6-9597-4950-a540-5eee05023666-tls-assets\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.208640 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.208551    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3951d4a6-9597-4950-a540-5eee05023666-config-out\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.208640 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.208600    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cspzz\" (UniqueName: \"kubernetes.io/projected/3951d4a6-9597-4950-a540-5eee05023666-kube-api-access-cspzz\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.208640 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.208639    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-kube-rbac-proxy\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.208919 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.208665    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-metrics-client-certs\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.208919 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.208694    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-grpc-tls\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.208919 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.208777    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-kubelet-serving-ca-bundle\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.208919 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.208806    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-2\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.208919 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.208804    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:30:36.208919 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.208831    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-db\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.211174 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.209687    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:30:36.211174 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.209750    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.211174 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.209762    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:30:36.211174 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.209786    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-metrics-client-ca\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.211174 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.209875    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-serving-certs-ca-bundle\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.211174 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.209877    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:30:36.211174 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.209902    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-trusted-ca-bundle\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.211174 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.209932    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.211174 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.210029    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:30:36.211174 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.210304    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:30:36.211174 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.210749    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:30:36.211174 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.211092    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-config\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.211174 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.211133    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:30:36.211174 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.211153    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-thanos-prometheus-http-client-file\") pod \"3951d4a6-9597-4950-a540-5eee05023666\" (UID: \"3951d4a6-9597-4950-a540-5eee05023666\") "
Sep 06 19:30:36.212040 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.211419    2571 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.212040 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.211438    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.212040 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.211453    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-db\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.212040 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.211466    2571 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-metrics-client-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.212040 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.211480    2571 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.212040 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.211496    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.212040 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.211510    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.212040 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.211526    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3951d4a6-9597-4950-a540-5eee05023666-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.212451 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.212391    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3951d4a6-9597-4950-a540-5eee05023666-kube-api-access-cspzz" (OuterVolumeSpecName: "kube-api-access-cspzz") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "kube-api-access-cspzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:30:36.212508 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.212473    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:36.212571 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.212510    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3951d4a6-9597-4950-a540-5eee05023666-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:30:36.212885 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.212851    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3951d4a6-9597-4950-a540-5eee05023666-config-out" (OuterVolumeSpecName: "config-out") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:30:36.213034 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.212910    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:36.213564 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.213537    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:36.213649 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.213569    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:36.213869 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.213848    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:36.214148 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.214110    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-config" (OuterVolumeSpecName: "config") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:36.214226 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.214190    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:36.214806 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.214785    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:36.222364 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.222337    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-web-config" (OuterVolumeSpecName: "web-config") pod "3951d4a6-9597-4950-a540-5eee05023666" (UID: "3951d4a6-9597-4950-a540-5eee05023666"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:30:36.311894 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.311839    2571 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-grpc-tls\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.311894 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.311890    2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.311894 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.311905    2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.312175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.311919    2571 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-config\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.312175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.311932    2571 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.312175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.311944    2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-web-config\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.312175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.311984    2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.312175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.311997    2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3951d4a6-9597-4950-a540-5eee05023666-tls-assets\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.312175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.312008    2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3951d4a6-9597-4950-a540-5eee05023666-config-out\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.312175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.312020    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cspzz\" (UniqueName: \"kubernetes.io/projected/3951d4a6-9597-4950-a540-5eee05023666-kube-api-access-cspzz\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.312175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.312031    2571 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-kube-rbac-proxy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.312175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.312043    2571 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3951d4a6-9597-4950-a540-5eee05023666-secret-metrics-client-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:30:36.836366 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.836330    2571 generic.go:356] "Generic (PLEG): container finished" podID="3951d4a6-9597-4950-a540-5eee05023666" containerID="bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c" exitCode=0
Sep 06 19:30:36.836366 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.836358    2571 generic.go:356] "Generic (PLEG): container finished" podID="3951d4a6-9597-4950-a540-5eee05023666" containerID="52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961" exitCode=0
Sep 06 19:30:36.836366 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.836365    2571 generic.go:356] "Generic (PLEG): container finished" podID="3951d4a6-9597-4950-a540-5eee05023666" containerID="a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106" exitCode=0
Sep 06 19:30:36.836366 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.836373    2571 generic.go:356] "Generic (PLEG): container finished" podID="3951d4a6-9597-4950-a540-5eee05023666" containerID="4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a" exitCode=0
Sep 06 19:30:36.836366 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.836378    2571 generic.go:356] "Generic (PLEG): container finished" podID="3951d4a6-9597-4950-a540-5eee05023666" containerID="a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2" exitCode=0
Sep 06 19:30:36.837012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.836384    2571 generic.go:356] "Generic (PLEG): container finished" podID="3951d4a6-9597-4950-a540-5eee05023666" containerID="99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4" exitCode=0
Sep 06 19:30:36.837012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.836411    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.837012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.836409    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3951d4a6-9597-4950-a540-5eee05023666","Type":"ContainerDied","Data":"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"}
Sep 06 19:30:36.837012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.836603    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3951d4a6-9597-4950-a540-5eee05023666","Type":"ContainerDied","Data":"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"}
Sep 06 19:30:36.837012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.836632    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3951d4a6-9597-4950-a540-5eee05023666","Type":"ContainerDied","Data":"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"}
Sep 06 19:30:36.837012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.836664    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3951d4a6-9597-4950-a540-5eee05023666","Type":"ContainerDied","Data":"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"}
Sep 06 19:30:36.837012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.836667    2571 scope.go:117] "RemoveContainer" containerID="bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"
Sep 06 19:30:36.837012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.836692    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3951d4a6-9597-4950-a540-5eee05023666","Type":"ContainerDied","Data":"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"}
Sep 06 19:30:36.837012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.836716    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3951d4a6-9597-4950-a540-5eee05023666","Type":"ContainerDied","Data":"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"}
Sep 06 19:30:36.837012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.836748    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3951d4a6-9597-4950-a540-5eee05023666","Type":"ContainerDied","Data":"8f0b8db2bff55df5ca4afba37e3399b99bd874d8fc9a7079b5d8592f18a55f27"}
Sep 06 19:30:36.837836 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.837813    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5" event={"ID":"81670dcf-0538-45c8-897d-91fd12a87bea","Type":"ContainerStarted","Data":"b9d723ee8a1e55888b126d0e6d8e5331a4c403c3fad1d19bd3a13750e65e9832"}
Sep 06 19:30:36.844686 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.844569    2571 scope.go:117] "RemoveContainer" containerID="52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"
Sep 06 19:30:36.852269 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.852113    2571 scope.go:117] "RemoveContainer" containerID="a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"
Sep 06 19:30:36.858345 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.858320    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 19:30:36.860568 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.860541    2571 scope.go:117] "RemoveContainer" containerID="4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"
Sep 06 19:30:36.861636 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.861616    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 19:30:36.868036 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.868008    2571 scope.go:117] "RemoveContainer" containerID="a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"
Sep 06 19:30:36.874874 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.874849    2571 scope.go:117] "RemoveContainer" containerID="99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"
Sep 06 19:30:36.882393 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.882372    2571 scope.go:117] "RemoveContainer" containerID="00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1"
Sep 06 19:30:36.882822 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.882804    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 19:30:36.883182 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883161    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="kube-rbac-proxy-thanos"
Sep 06 19:30:36.883286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883183    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="kube-rbac-proxy-thanos"
Sep 06 19:30:36.883286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883201    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="thanos-sidecar"
Sep 06 19:30:36.883286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883210    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="thanos-sidecar"
Sep 06 19:30:36.883286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883223    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="kube-rbac-proxy"
Sep 06 19:30:36.883286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883229    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="kube-rbac-proxy"
Sep 06 19:30:36.883286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883236    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="init-config-reloader"
Sep 06 19:30:36.883286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883244    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="init-config-reloader"
Sep 06 19:30:36.883286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883251    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="kube-rbac-proxy-web"
Sep 06 19:30:36.883286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883258    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="kube-rbac-proxy-web"
Sep 06 19:30:36.883286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883267    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="prometheus"
Sep 06 19:30:36.883286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883273    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="prometheus"
Sep 06 19:30:36.883286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883281    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="config-reloader"
Sep 06 19:30:36.883286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883290    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="config-reloader"
Sep 06 19:30:36.883688 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883379    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="kube-rbac-proxy"
Sep 06 19:30:36.883688 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883390    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="config-reloader"
Sep 06 19:30:36.883688 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883398    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="prometheus"
Sep 06 19:30:36.883688 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883405    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="kube-rbac-proxy-web"
Sep 06 19:30:36.883688 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883414    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="thanos-sidecar"
Sep 06 19:30:36.883688 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.883421    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3951d4a6-9597-4950-a540-5eee05023666" containerName="kube-rbac-proxy-thanos"
Sep 06 19:30:36.887410 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.887389    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.890514 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.890379    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 06 19:30:36.890695 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.890666    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4wd79\""
Sep 06 19:30:36.890695 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.890678    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 06 19:30:36.890848 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.890828    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 06 19:30:36.891037 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.890923    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7m11301d8hlnm\""
Sep 06 19:30:36.891452 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.891414    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 06 19:30:36.891786 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.891768    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 06 19:30:36.891972 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.891795    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 06 19:30:36.892056 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.891813    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 06 19:30:36.892056 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.891814    2571 scope.go:117] "RemoveContainer" containerID="bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"
Sep 06 19:30:36.892155 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.891823    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 06 19:30:36.892155 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.891831    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 06 19:30:36.892155 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.891838    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 06 19:30:36.892297 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.891761    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 06 19:30:36.892297 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.891877    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 06 19:30:36.892471 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:30:36.892452    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c\": container with ID starting with bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c not found: ID does not exist" containerID="bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"
Sep 06 19:30:36.892546 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.892482    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"} err="failed to get container status \"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c\": rpc error: code = NotFound desc = could not find container \"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c\": container with ID starting with bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c not found: ID does not exist"
Sep 06 19:30:36.892546 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.892499    2571 scope.go:117] "RemoveContainer" containerID="52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"
Sep 06 19:30:36.892822 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:30:36.892796    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961\": container with ID starting with 52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961 not found: ID does not exist" containerID="52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"
Sep 06 19:30:36.892875 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.892832    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"} err="failed to get container status \"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961\": rpc error: code = NotFound desc = could not find container \"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961\": container with ID starting with 52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961 not found: ID does not exist"
Sep 06 19:30:36.892875 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.892855    2571 scope.go:117] "RemoveContainer" containerID="a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"
Sep 06 19:30:36.893182 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:30:36.893158    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106\": container with ID starting with a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106 not found: ID does not exist" containerID="a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"
Sep 06 19:30:36.893630 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.893597    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"} err="failed to get container status \"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106\": rpc error: code = NotFound desc = could not find container \"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106\": container with ID starting with a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106 not found: ID does not exist"
Sep 06 19:30:36.893772 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.893749    2571 scope.go:117] "RemoveContainer" containerID="4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"
Sep 06 19:30:36.894253 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:30:36.894228    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a\": container with ID starting with 4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a not found: ID does not exist" containerID="4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"
Sep 06 19:30:36.894336 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.894262    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"} err="failed to get container status \"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a\": rpc error: code = NotFound desc = could not find container \"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a\": container with ID starting with 4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a not found: ID does not exist"
Sep 06 19:30:36.894336 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.894284    2571 scope.go:117] "RemoveContainer" containerID="a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"
Sep 06 19:30:36.894847 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:30:36.894817    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2\": container with ID starting with a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2 not found: ID does not exist" containerID="a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"
Sep 06 19:30:36.894926 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.894855    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"} err="failed to get container status \"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2\": rpc error: code = NotFound desc = could not find container \"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2\": container with ID starting with a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2 not found: ID does not exist"
Sep 06 19:30:36.894926 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.894876    2571 scope.go:117] "RemoveContainer" containerID="99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"
Sep 06 19:30:36.897025 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.896997    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 06 19:30:36.897371 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:30:36.897326    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4\": container with ID starting with 99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4 not found: ID does not exist" containerID="99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"
Sep 06 19:30:36.897371 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.897354    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"} err="failed to get container status \"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4\": rpc error: code = NotFound desc = could not find container \"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4\": container with ID starting with 99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4 not found: ID does not exist"
Sep 06 19:30:36.897509 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.897376    2571 scope.go:117] "RemoveContainer" containerID="00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1"
Sep 06 19:30:36.897773 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:30:36.897748    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1\": container with ID starting with 00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1 not found: ID does not exist" containerID="00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1"
Sep 06 19:30:36.897902 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.897879    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1"} err="failed to get container status \"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1\": rpc error: code = NotFound desc = could not find container \"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1\": container with ID starting with 00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1 not found: ID does not exist"
Sep 06 19:30:36.898010 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.897998    2571 scope.go:117] "RemoveContainer" containerID="bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"
Sep 06 19:30:36.900362 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.900073    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"} err="failed to get container status \"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c\": rpc error: code = NotFound desc = could not find container \"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c\": container with ID starting with bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c not found: ID does not exist"
Sep 06 19:30:36.900362 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.900100    2571 scope.go:117] "RemoveContainer" containerID="52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"
Sep 06 19:30:36.901473 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.901441    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 06 19:30:36.901572 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.901437    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"} err="failed to get container status \"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961\": rpc error: code = NotFound desc = could not find container \"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961\": container with ID starting with 52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961 not found: ID does not exist"
Sep 06 19:30:36.901572 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.901510    2571 scope.go:117] "RemoveContainer" containerID="a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"
Sep 06 19:30:36.902283 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.902225    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"} err="failed to get container status \"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106\": rpc error: code = NotFound desc = could not find container \"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106\": container with ID starting with a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106 not found: ID does not exist"
Sep 06 19:30:36.902283 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.902256    2571 scope.go:117] "RemoveContainer" containerID="4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"
Sep 06 19:30:36.902603 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.902578    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"} err="failed to get container status \"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a\": rpc error: code = NotFound desc = could not find container \"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a\": container with ID starting with 4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a not found: ID does not exist"
Sep 06 19:30:36.902695 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.902607    2571 scope.go:117] "RemoveContainer" containerID="a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"
Sep 06 19:30:36.903908 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.903770    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"} err="failed to get container status \"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2\": rpc error: code = NotFound desc = could not find container \"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2\": container with ID starting with a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2 not found: ID does not exist"
Sep 06 19:30:36.903908 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.903799    2571 scope.go:117] "RemoveContainer" containerID="99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"
Sep 06 19:30:36.904474 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.904450    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 19:30:36.904582 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.904486    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"} err="failed to get container status \"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4\": rpc error: code = NotFound desc = could not find container \"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4\": container with ID starting with 99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4 not found: ID does not exist"
Sep 06 19:30:36.904582 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.904509    2571 scope.go:117] "RemoveContainer" containerID="00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1"
Sep 06 19:30:36.904802 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.904776    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1"} err="failed to get container status \"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1\": rpc error: code = NotFound desc = could not find container \"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1\": container with ID starting with 00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1 not found: ID does not exist"
Sep 06 19:30:36.904855 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.904805    2571 scope.go:117] "RemoveContainer" containerID="bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"
Sep 06 19:30:36.905122 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.905100    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"} err="failed to get container status \"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c\": rpc error: code = NotFound desc = could not find container \"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c\": container with ID starting with bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c not found: ID does not exist"
Sep 06 19:30:36.905192 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.905125    2571 scope.go:117] "RemoveContainer" containerID="52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"
Sep 06 19:30:36.905442 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.905420    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"} err="failed to get container status \"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961\": rpc error: code = NotFound desc = could not find container \"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961\": container with ID starting with 52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961 not found: ID does not exist"
Sep 06 19:30:36.905498 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.905444    2571 scope.go:117] "RemoveContainer" containerID="a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"
Sep 06 19:30:36.905756 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.905735    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"} err="failed to get container status \"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106\": rpc error: code = NotFound desc = could not find container \"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106\": container with ID starting with a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106 not found: ID does not exist"
Sep 06 19:30:36.905823 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.905757    2571 scope.go:117] "RemoveContainer" containerID="4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"
Sep 06 19:30:36.906082 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.906061    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"} err="failed to get container status \"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a\": rpc error: code = NotFound desc = could not find container \"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a\": container with ID starting with 4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a not found: ID does not exist"
Sep 06 19:30:36.906173 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.906084    2571 scope.go:117] "RemoveContainer" containerID="a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"
Sep 06 19:30:36.906369 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.906350    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"} err="failed to get container status \"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2\": rpc error: code = NotFound desc = could not find container \"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2\": container with ID starting with a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2 not found: ID does not exist"
Sep 06 19:30:36.906369 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.906368    2571 scope.go:117] "RemoveContainer" containerID="99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"
Sep 06 19:30:36.906615 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.906598    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"} err="failed to get container status \"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4\": rpc error: code = NotFound desc = could not find container \"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4\": container with ID starting with 99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4 not found: ID does not exist"
Sep 06 19:30:36.906615 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.906615    2571 scope.go:117] "RemoveContainer" containerID="00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1"
Sep 06 19:30:36.906848 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.906825    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1"} err="failed to get container status \"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1\": rpc error: code = NotFound desc = could not find container \"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1\": container with ID starting with 00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1 not found: ID does not exist"
Sep 06 19:30:36.906893 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.906852    2571 scope.go:117] "RemoveContainer" containerID="bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"
Sep 06 19:30:36.907159 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.907141    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"} err="failed to get container status \"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c\": rpc error: code = NotFound desc = could not find container \"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c\": container with ID starting with bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c not found: ID does not exist"
Sep 06 19:30:36.907159 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.907159    2571 scope.go:117] "RemoveContainer" containerID="52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"
Sep 06 19:30:36.907417 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.907399    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"} err="failed to get container status \"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961\": rpc error: code = NotFound desc = could not find container \"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961\": container with ID starting with 52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961 not found: ID does not exist"
Sep 06 19:30:36.907462 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.907418    2571 scope.go:117] "RemoveContainer" containerID="a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"
Sep 06 19:30:36.907663 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.907641    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"} err="failed to get container status \"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106\": rpc error: code = NotFound desc = could not find container \"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106\": container with ID starting with a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106 not found: ID does not exist"
Sep 06 19:30:36.907663 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.907662    2571 scope.go:117] "RemoveContainer" containerID="4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"
Sep 06 19:30:36.907947 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.907922    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"} err="failed to get container status \"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a\": rpc error: code = NotFound desc = could not find container \"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a\": container with ID starting with 4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a not found: ID does not exist"
Sep 06 19:30:36.908024 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.907948    2571 scope.go:117] "RemoveContainer" containerID="a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"
Sep 06 19:30:36.908236 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.908217    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"} err="failed to get container status \"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2\": rpc error: code = NotFound desc = could not find container \"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2\": container with ID starting with a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2 not found: ID does not exist"
Sep 06 19:30:36.908284 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.908237    2571 scope.go:117] "RemoveContainer" containerID="99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"
Sep 06 19:30:36.908476 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.908456    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"} err="failed to get container status \"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4\": rpc error: code = NotFound desc = could not find container \"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4\": container with ID starting with 99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4 not found: ID does not exist"
Sep 06 19:30:36.908476 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.908474    2571 scope.go:117] "RemoveContainer" containerID="00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1"
Sep 06 19:30:36.908710 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.908689    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1"} err="failed to get container status \"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1\": rpc error: code = NotFound desc = could not find container \"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1\": container with ID starting with 00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1 not found: ID does not exist"
Sep 06 19:30:36.908710 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.908710    2571 scope.go:117] "RemoveContainer" containerID="bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"
Sep 06 19:30:36.908933 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.908915    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"} err="failed to get container status \"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c\": rpc error: code = NotFound desc = could not find container \"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c\": container with ID starting with bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c not found: ID does not exist"
Sep 06 19:30:36.909007 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.908934    2571 scope.go:117] "RemoveContainer" containerID="52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"
Sep 06 19:30:36.909177 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.909157    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"} err="failed to get container status \"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961\": rpc error: code = NotFound desc = could not find container \"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961\": container with ID starting with 52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961 not found: ID does not exist"
Sep 06 19:30:36.909229 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.909177    2571 scope.go:117] "RemoveContainer" containerID="a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"
Sep 06 19:30:36.909390 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.909373    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"} err="failed to get container status \"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106\": rpc error: code = NotFound desc = could not find container \"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106\": container with ID starting with a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106 not found: ID does not exist"
Sep 06 19:30:36.909436 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.909392    2571 scope.go:117] "RemoveContainer" containerID="4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"
Sep 06 19:30:36.909663 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.909646    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"} err="failed to get container status \"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a\": rpc error: code = NotFound desc = could not find container \"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a\": container with ID starting with 4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a not found: ID does not exist"
Sep 06 19:30:36.909712 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.909663    2571 scope.go:117] "RemoveContainer" containerID="a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"
Sep 06 19:30:36.909894 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.909874    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"} err="failed to get container status \"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2\": rpc error: code = NotFound desc = could not find container \"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2\": container with ID starting with a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2 not found: ID does not exist"
Sep 06 19:30:36.909933 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.909894    2571 scope.go:117] "RemoveContainer" containerID="99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"
Sep 06 19:30:36.910164 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.910146    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"} err="failed to get container status \"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4\": rpc error: code = NotFound desc = could not find container \"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4\": container with ID starting with 99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4 not found: ID does not exist"
Sep 06 19:30:36.910228 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.910168    2571 scope.go:117] "RemoveContainer" containerID="00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1"
Sep 06 19:30:36.910415 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.910398    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1"} err="failed to get container status \"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1\": rpc error: code = NotFound desc = could not find container \"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1\": container with ID starting with 00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1 not found: ID does not exist"
Sep 06 19:30:36.910467 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.910416    2571 scope.go:117] "RemoveContainer" containerID="bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"
Sep 06 19:30:36.910672 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.910654    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c"} err="failed to get container status \"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c\": rpc error: code = NotFound desc = could not find container \"bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c\": container with ID starting with bc3e5a4bd54b99da1bcaf7026565338ab416c1e704b956c8aa7363d96e445c8c not found: ID does not exist"
Sep 06 19:30:36.910739 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.910673    2571 scope.go:117] "RemoveContainer" containerID="52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"
Sep 06 19:30:36.910910 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.910893    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961"} err="failed to get container status \"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961\": rpc error: code = NotFound desc = could not find container \"52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961\": container with ID starting with 52c2c6a3a40659f7058603c591abced782ea1a16d93bd00f1373e56aa31ac961 not found: ID does not exist"
Sep 06 19:30:36.910952 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.910910    2571 scope.go:117] "RemoveContainer" containerID="a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"
Sep 06 19:30:36.911248 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.911222    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106"} err="failed to get container status \"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106\": rpc error: code = NotFound desc = could not find container \"a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106\": container with ID starting with a47fdce2c392543ebeef56574a25096a300e68e1587ae8dfa715c4bb6e2f8106 not found: ID does not exist"
Sep 06 19:30:36.911248 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.911247    2571 scope.go:117] "RemoveContainer" containerID="4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"
Sep 06 19:30:36.911568 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.911531    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a"} err="failed to get container status \"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a\": rpc error: code = NotFound desc = could not find container \"4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a\": container with ID starting with 4023322609a48a48f7acf26429461a75af6f8044c84aa105fd297bc057a73b4a not found: ID does not exist"
Sep 06 19:30:36.911568 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.911567    2571 scope.go:117] "RemoveContainer" containerID="a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"
Sep 06 19:30:36.912003 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.911912    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2"} err="failed to get container status \"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2\": rpc error: code = NotFound desc = could not find container \"a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2\": container with ID starting with a82d26fc868a17cc1fd06f73544917951778b5e8d38e2e6db92ad711a94103a2 not found: ID does not exist"
Sep 06 19:30:36.912003 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.911939    2571 scope.go:117] "RemoveContainer" containerID="99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"
Sep 06 19:30:36.912346 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.912327    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4"} err="failed to get container status \"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4\": rpc error: code = NotFound desc = could not find container \"99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4\": container with ID starting with 99d3862fa415b4d3984af99387cc49f81a0c584124fef9e9f9d691744fa48ba4 not found: ID does not exist"
Sep 06 19:30:36.912414 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.912347    2571 scope.go:117] "RemoveContainer" containerID="00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1"
Sep 06 19:30:36.912657 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.912635    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1"} err="failed to get container status \"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1\": rpc error: code = NotFound desc = could not find container \"00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1\": container with ID starting with 00e9327498168a532fd539c69a4812fd175608f752bfca841cd90dab88164cf1 not found: ID does not exist"
Sep 06 19:30:36.916343 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.916322    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.916448 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.916378    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.916511 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.916478    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-web-config\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.916562 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.916510    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65d0c889-3434-4cf5-87a4-6ae31842fd29-config-out\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.916562 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.916538    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.916666 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.916562    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65d0c889-3434-4cf5-87a4-6ae31842fd29-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.916666 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.916604    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.916666 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.916639    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.916817 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.916707    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.916892 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.916860    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.917047 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.917026    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-config\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.917116 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.917060    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.917116 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.917088    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.917218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.917143    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.917218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.917170    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65d0c889-3434-4cf5-87a4-6ae31842fd29-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.917218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.917207    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.917353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.917242    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.917353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.917265    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.917353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.917322    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.917465 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.917351    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-822fx\" (UniqueName: \"kubernetes.io/projected/65d0c889-3434-4cf5-87a4-6ae31842fd29-kube-api-access-822fx\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:36.917601 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:36.917578    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3951d4a6-9597-4950-a540-5eee05023666" path="/var/lib/kubelet/pods/3951d4a6-9597-4950-a540-5eee05023666/volumes"
Sep 06 19:30:37.018255 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018213    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.018255 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018257    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65d0c889-3434-4cf5-87a4-6ae31842fd29-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.018551 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018278    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.018551 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018408    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.018551 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018460    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.018551 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018513    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.018551 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018544    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-822fx\" (UniqueName: \"kubernetes.io/projected/65d0c889-3434-4cf5-87a4-6ae31842fd29-kube-api-access-822fx\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.018861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018588    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.018861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018626    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.018861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018657    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-web-config\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.018861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018683    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65d0c889-3434-4cf5-87a4-6ae31842fd29-config-out\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.018861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018706    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.018861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018728    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65d0c889-3434-4cf5-87a4-6ae31842fd29-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.018861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018768    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.018861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018801    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.018861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018838    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.019420 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018873    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.019420 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018917    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-config\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.019420 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018941    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.019420 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.018996    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.019420 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.019030    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.019420 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.019090    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.019857 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.019546    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.020209 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.020101    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.021063 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.020522    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65d0c889-3434-4cf5-87a4-6ae31842fd29-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.021063 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.020762    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.023789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.021210    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.023789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.022104    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65d0c889-3434-4cf5-87a4-6ae31842fd29-config-out\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.023789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.022371    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65d0c889-3434-4cf5-87a4-6ae31842fd29-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.023789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.022644    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.023789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.022822    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-web-config\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.023789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.023377    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.024264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.024045    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.024264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.024201    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-config\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.024792 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.024762    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65d0c889-3434-4cf5-87a4-6ae31842fd29-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.024921 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.024823    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.024921 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.024860    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.025083 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.024932    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.025149 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.025134    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65d0c889-3434-4cf5-87a4-6ae31842fd29-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.026889 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.026871    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-822fx\" (UniqueName: \"kubernetes.io/projected/65d0c889-3434-4cf5-87a4-6ae31842fd29-kube-api-access-822fx\") pod \"prometheus-k8s-0\" (UID: \"65d0c889-3434-4cf5-87a4-6ae31842fd29\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.207559 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.207467    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:30:37.493344 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.493256    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 19:30:37.496875 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:30:37.496843    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d0c889_3434_4cf5_87a4_6ae31842fd29.slice/crio-aec61a2f7bc2ccdc9094dd95d9af0b98537566e4f5d3db2c5e12459b27e7c7f1 WatchSource:0}: Error finding container aec61a2f7bc2ccdc9094dd95d9af0b98537566e4f5d3db2c5e12459b27e7c7f1: Status 404 returned error can't find the container with id aec61a2f7bc2ccdc9094dd95d9af0b98537566e4f5d3db2c5e12459b27e7c7f1
Sep 06 19:30:37.844704 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.844664    2571 generic.go:356] "Generic (PLEG): container finished" podID="65d0c889-3434-4cf5-87a4-6ae31842fd29" containerID="b417f6885a4ee3cdd37d39192a3946e18b98311054aa2df4177bcf5f66f79460" exitCode=0
Sep 06 19:30:37.845205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.844758    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65d0c889-3434-4cf5-87a4-6ae31842fd29","Type":"ContainerDied","Data":"b417f6885a4ee3cdd37d39192a3946e18b98311054aa2df4177bcf5f66f79460"}
Sep 06 19:30:37.845205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:37.844833    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65d0c889-3434-4cf5-87a4-6ae31842fd29","Type":"ContainerStarted","Data":"aec61a2f7bc2ccdc9094dd95d9af0b98537566e4f5d3db2c5e12459b27e7c7f1"}
Sep 06 19:30:38.855225 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:38.855181    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5" event={"ID":"81670dcf-0538-45c8-897d-91fd12a87bea","Type":"ContainerStarted","Data":"4c800576dfa5e67443a51533acab7bfc5b314582a04617275bf2c5bddc7db194"}
Sep 06 19:30:38.855680 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:38.855284    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5" event={"ID":"81670dcf-0538-45c8-897d-91fd12a87bea","Type":"ContainerStarted","Data":"38c22769a340e8af87d253c4465073ae1e07cd748bc1e09ab6a8b834e135ee07"}
Sep 06 19:30:38.855680 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:38.855306    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5" event={"ID":"81670dcf-0538-45c8-897d-91fd12a87bea","Type":"ContainerStarted","Data":"8ae5acfb3a8b5a42d22cf150fd4cd51dd956b3ece3c467ebb9fa9a5ecec58d46"}
Sep 06 19:30:38.858564 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:38.858534    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65d0c889-3434-4cf5-87a4-6ae31842fd29","Type":"ContainerStarted","Data":"ee5faa847a68627e46de3530f3ac8b0b4bbb1d96afe63ad0a810233ae852b497"}
Sep 06 19:30:38.858714 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:38.858613    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65d0c889-3434-4cf5-87a4-6ae31842fd29","Type":"ContainerStarted","Data":"668b897fccaee72e933621f171e915b0290416df8af147d49f4b19aa6c3eab34"}
Sep 06 19:30:38.858714 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:38.858644    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65d0c889-3434-4cf5-87a4-6ae31842fd29","Type":"ContainerStarted","Data":"6b9b86e06374fe32a51f0bd614a75e9579af29272b3066da57b2f1da5c521cbe"}
Sep 06 19:30:38.858714 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:38.858669    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65d0c889-3434-4cf5-87a4-6ae31842fd29","Type":"ContainerStarted","Data":"17aa5918180ace55774919d6965bbb8e5fe366f8187118dce51565d96d68b8a9"}
Sep 06 19:30:38.858714 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:38.858694    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65d0c889-3434-4cf5-87a4-6ae31842fd29","Type":"ContainerStarted","Data":"f5ecac9c15c296ff3f83f40d3183a4e198eace1203cecc221928daa9ecb091fb"}
Sep 06 19:30:38.858934 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:38.858717    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65d0c889-3434-4cf5-87a4-6ae31842fd29","Type":"ContainerStarted","Data":"e92c196ad6ebacd11cef3a126bba9da4a451595701c099b4dfab7f686568a205"}
Sep 06 19:30:38.876537 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:38.876473    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6df9f98f87-jgzb5" podStartSLOduration=1.856647255 podStartE2EDuration="3.876453485s" totalImagesPullingTime="2.01980623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:30:38.874640823 +0000 UTC m=+202.444939169" watchObservedRunningTime="2026-09-06 19:30:38.876453485 +0000 UTC m=+202.446751831"
Sep 06 19:30:38.898865 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:38.898813    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.87677479 podStartE2EDuration="2.898795559s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.020769ms" isStatefulPod=false podCreationTimestamp="2026-09-06 19:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:30:38.897379647 +0000 UTC m=+202.467678033" watchObservedRunningTime="2026-09-06 19:30:38.898795559 +0000 UTC m=+202.469093905"
Sep 06 19:30:42.208358 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:30:42.208308    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:31:37.207930 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:31:37.207868    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:31:37.224031 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:31:37.223926    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:31:38.106874 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:31:38.106847    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 19:32:16.828995 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:32:16.828947    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 19:32:16.829562 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:32:16.829083    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 19:32:16.839205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:32:16.839182    2571 kubelet.go:1648] "Image garbage collection succeeded"
Sep 06 19:33:06.435048 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:06.435006    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xghb7"]
Sep 06 19:33:06.438616 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:06.438592    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xghb7"
Sep 06 19:33:06.440794 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:06.440763    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 06 19:33:06.441765 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:06.441743    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-m295h\""
Sep 06 19:33:06.441878 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:06.441773    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 06 19:33:06.448749 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:06.448725    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xghb7"]
Sep 06 19:33:06.611891 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:06.611848    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db7ed180-85d2-4d30-b8a7-3174d8f28e82-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xghb7\" (UID: \"db7ed180-85d2-4d30-b8a7-3174d8f28e82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xghb7"
Sep 06 19:33:06.612103 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:06.611900    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr2mq\" (UniqueName: \"kubernetes.io/projected/db7ed180-85d2-4d30-b8a7-3174d8f28e82-kube-api-access-fr2mq\") pod \"cert-manager-webhook-6486ffd997-xghb7\" (UID: \"db7ed180-85d2-4d30-b8a7-3174d8f28e82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xghb7"
Sep 06 19:33:06.712782 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:06.712693    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db7ed180-85d2-4d30-b8a7-3174d8f28e82-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xghb7\" (UID: \"db7ed180-85d2-4d30-b8a7-3174d8f28e82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xghb7"
Sep 06 19:33:06.712782 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:06.712747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr2mq\" (UniqueName: \"kubernetes.io/projected/db7ed180-85d2-4d30-b8a7-3174d8f28e82-kube-api-access-fr2mq\") pod \"cert-manager-webhook-6486ffd997-xghb7\" (UID: \"db7ed180-85d2-4d30-b8a7-3174d8f28e82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xghb7"
Sep 06 19:33:06.720350 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:06.720317    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db7ed180-85d2-4d30-b8a7-3174d8f28e82-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xghb7\" (UID: \"db7ed180-85d2-4d30-b8a7-3174d8f28e82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xghb7"
Sep 06 19:33:06.720480 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:06.720461    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr2mq\" (UniqueName: \"kubernetes.io/projected/db7ed180-85d2-4d30-b8a7-3174d8f28e82-kube-api-access-fr2mq\") pod \"cert-manager-webhook-6486ffd997-xghb7\" (UID: \"db7ed180-85d2-4d30-b8a7-3174d8f28e82\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xghb7"
Sep 06 19:33:06.748524 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:06.748491    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xghb7"
Sep 06 19:33:07.017013 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:07.016907    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xghb7"]
Sep 06 19:33:07.020008 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:33:07.019978    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb7ed180_85d2_4d30_b8a7_3174d8f28e82.slice/crio-66979f8ebbeb5d68e359a2352509381e94a238b5f1a29c4e1e2b8a8610e20eb2 WatchSource:0}: Error finding container 66979f8ebbeb5d68e359a2352509381e94a238b5f1a29c4e1e2b8a8610e20eb2: Status 404 returned error can't find the container with id 66979f8ebbeb5d68e359a2352509381e94a238b5f1a29c4e1e2b8a8610e20eb2
Sep 06 19:33:07.021975 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:07.021938    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 19:33:07.449805 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:07.449763    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xghb7" event={"ID":"db7ed180-85d2-4d30-b8a7-3174d8f28e82","Type":"ContainerStarted","Data":"66979f8ebbeb5d68e359a2352509381e94a238b5f1a29c4e1e2b8a8610e20eb2"}
Sep 06 19:33:10.470215 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:10.470120    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xghb7" event={"ID":"db7ed180-85d2-4d30-b8a7-3174d8f28e82","Type":"ContainerStarted","Data":"48a8f61304558c957db03012b40c55b06ac55ad7709ea339cce0dd7e67bec6fd"}
Sep 06 19:33:10.470215 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:10.470192    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xghb7"
Sep 06 19:33:10.486090 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:10.486040    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-xghb7" podStartSLOduration=1.301416413 podStartE2EDuration="4.486025533s" totalImagesPullingTime="3.18460912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:33:10.484665895 +0000 UTC m=+354.054964241" watchObservedRunningTime="2026-09-06 19:33:10.486025533 +0000 UTC m=+354.056323879"
Sep 06 19:33:16.476792 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:16.476760    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xghb7"
Sep 06 19:33:38.545554 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.545513    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"]
Sep 06 19:33:38.550388 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.550362    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.553008 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.552978    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 06 19:33:38.553153 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.552979    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 06 19:33:38.553153 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.553020    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-pp5zq\""
Sep 06 19:33:38.553153 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.553119    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 06 19:33:38.554078 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.554059    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 06 19:33:38.565141 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.565110    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"]
Sep 06 19:33:38.679922 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.679875    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bec8c6aa-d55b-4505-9f46-740144961193-tmp\") pod \"opendatahub-operator-controller-manager-7f964dcc99-npwhb\" (UID: \"bec8c6aa-d55b-4505-9f46-740144961193\") " pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.680139 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.679954    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bec8c6aa-d55b-4505-9f46-740144961193-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f964dcc99-npwhb\" (UID: \"bec8c6aa-d55b-4505-9f46-740144961193\") " pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.680139 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.680025    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bec8c6aa-d55b-4505-9f46-740144961193-manifests\") pod \"opendatahub-operator-controller-manager-7f964dcc99-npwhb\" (UID: \"bec8c6aa-d55b-4505-9f46-740144961193\") " pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.680139 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.680042    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bec8c6aa-d55b-4505-9f46-740144961193-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f964dcc99-npwhb\" (UID: \"bec8c6aa-d55b-4505-9f46-740144961193\") " pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.680139 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.680119    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfh6p\" (UniqueName: \"kubernetes.io/projected/bec8c6aa-d55b-4505-9f46-740144961193-kube-api-access-wfh6p\") pod \"opendatahub-operator-controller-manager-7f964dcc99-npwhb\" (UID: \"bec8c6aa-d55b-4505-9f46-740144961193\") " pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.781468 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.781427    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bec8c6aa-d55b-4505-9f46-740144961193-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f964dcc99-npwhb\" (UID: \"bec8c6aa-d55b-4505-9f46-740144961193\") " pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.781638 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.781488    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bec8c6aa-d55b-4505-9f46-740144961193-manifests\") pod \"opendatahub-operator-controller-manager-7f964dcc99-npwhb\" (UID: \"bec8c6aa-d55b-4505-9f46-740144961193\") " pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.781638 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.781503    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bec8c6aa-d55b-4505-9f46-740144961193-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f964dcc99-npwhb\" (UID: \"bec8c6aa-d55b-4505-9f46-740144961193\") " pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.781638 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.781526    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfh6p\" (UniqueName: \"kubernetes.io/projected/bec8c6aa-d55b-4505-9f46-740144961193-kube-api-access-wfh6p\") pod \"opendatahub-operator-controller-manager-7f964dcc99-npwhb\" (UID: \"bec8c6aa-d55b-4505-9f46-740144961193\") " pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.781638 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.781554    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bec8c6aa-d55b-4505-9f46-740144961193-tmp\") pod \"opendatahub-operator-controller-manager-7f964dcc99-npwhb\" (UID: \"bec8c6aa-d55b-4505-9f46-740144961193\") " pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.781970 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.781939    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bec8c6aa-d55b-4505-9f46-740144961193-manifests\") pod \"opendatahub-operator-controller-manager-7f964dcc99-npwhb\" (UID: \"bec8c6aa-d55b-4505-9f46-740144961193\") " pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.783953 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.783922    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bec8c6aa-d55b-4505-9f46-740144961193-tmp\") pod \"opendatahub-operator-controller-manager-7f964dcc99-npwhb\" (UID: \"bec8c6aa-d55b-4505-9f46-740144961193\") " pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.784110 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.784090    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bec8c6aa-d55b-4505-9f46-740144961193-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f964dcc99-npwhb\" (UID: \"bec8c6aa-d55b-4505-9f46-740144961193\") " pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.784219 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.784198    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bec8c6aa-d55b-4505-9f46-740144961193-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f964dcc99-npwhb\" (UID: \"bec8c6aa-d55b-4505-9f46-740144961193\") " pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.789582 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.789550    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfh6p\" (UniqueName: \"kubernetes.io/projected/bec8c6aa-d55b-4505-9f46-740144961193-kube-api-access-wfh6p\") pod \"opendatahub-operator-controller-manager-7f964dcc99-npwhb\" (UID: \"bec8c6aa-d55b-4505-9f46-740144961193\") " pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:38.861769 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:38.861727    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:39.121214 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.121131    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"]
Sep 06 19:33:39.123788 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.123766    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:39.126444 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.126418    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 06 19:33:39.126685 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.126669    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 06 19:33:39.126992 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.126972    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 06 19:33:39.127753 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.127725    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 06 19:33:39.127854 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.127786    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 06 19:33:39.127912 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.127865    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9mh7l\""
Sep 06 19:33:39.139138 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.139106    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"]
Sep 06 19:33:39.143781 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.143751    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"]
Sep 06 19:33:39.146715 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:33:39.146675    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec8c6aa_d55b_4505_9f46_740144961193.slice/crio-8e3689304da4bbd683555218168169f783f68dd5e6d206c668154aedb7f49f0b WatchSource:0}: Error finding container 8e3689304da4bbd683555218168169f783f68dd5e6d206c668154aedb7f49f0b: Status 404 returned error can't find the container with id 8e3689304da4bbd683555218168169f783f68dd5e6d206c668154aedb7f49f0b
Sep 06 19:33:39.185014 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.184979    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6jtm\" (UniqueName: \"kubernetes.io/projected/03a1c734-f7c7-45d2-9868-eb943d7f28c9-kube-api-access-j6jtm\") pod \"lws-controller-manager-7769675c48-rn4w6\" (UID: \"03a1c734-f7c7-45d2-9868-eb943d7f28c9\") " pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:39.185198 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.185044    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/03a1c734-f7c7-45d2-9868-eb943d7f28c9-metrics-cert\") pod \"lws-controller-manager-7769675c48-rn4w6\" (UID: \"03a1c734-f7c7-45d2-9868-eb943d7f28c9\") " pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:39.185198 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.185073    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03a1c734-f7c7-45d2-9868-eb943d7f28c9-cert\") pod \"lws-controller-manager-7769675c48-rn4w6\" (UID: \"03a1c734-f7c7-45d2-9868-eb943d7f28c9\") " pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:39.185198 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.185106    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/03a1c734-f7c7-45d2-9868-eb943d7f28c9-manager-config\") pod \"lws-controller-manager-7769675c48-rn4w6\" (UID: \"03a1c734-f7c7-45d2-9868-eb943d7f28c9\") " pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:39.286293 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.286256    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/03a1c734-f7c7-45d2-9868-eb943d7f28c9-metrics-cert\") pod \"lws-controller-manager-7769675c48-rn4w6\" (UID: \"03a1c734-f7c7-45d2-9868-eb943d7f28c9\") " pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:39.286457 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.286307    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03a1c734-f7c7-45d2-9868-eb943d7f28c9-cert\") pod \"lws-controller-manager-7769675c48-rn4w6\" (UID: \"03a1c734-f7c7-45d2-9868-eb943d7f28c9\") " pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:39.286457 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.286362    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/03a1c734-f7c7-45d2-9868-eb943d7f28c9-manager-config\") pod \"lws-controller-manager-7769675c48-rn4w6\" (UID: \"03a1c734-f7c7-45d2-9868-eb943d7f28c9\") " pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:39.286457 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.286436    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6jtm\" (UniqueName: \"kubernetes.io/projected/03a1c734-f7c7-45d2-9868-eb943d7f28c9-kube-api-access-j6jtm\") pod \"lws-controller-manager-7769675c48-rn4w6\" (UID: \"03a1c734-f7c7-45d2-9868-eb943d7f28c9\") " pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:39.287127 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.287101    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/03a1c734-f7c7-45d2-9868-eb943d7f28c9-manager-config\") pod \"lws-controller-manager-7769675c48-rn4w6\" (UID: \"03a1c734-f7c7-45d2-9868-eb943d7f28c9\") " pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:39.288866 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.288841    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03a1c734-f7c7-45d2-9868-eb943d7f28c9-cert\") pod \"lws-controller-manager-7769675c48-rn4w6\" (UID: \"03a1c734-f7c7-45d2-9868-eb943d7f28c9\") " pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:39.288993 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.288945    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/03a1c734-f7c7-45d2-9868-eb943d7f28c9-metrics-cert\") pod \"lws-controller-manager-7769675c48-rn4w6\" (UID: \"03a1c734-f7c7-45d2-9868-eb943d7f28c9\") " pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:39.294495 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.294469    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6jtm\" (UniqueName: \"kubernetes.io/projected/03a1c734-f7c7-45d2-9868-eb943d7f28c9-kube-api-access-j6jtm\") pod \"lws-controller-manager-7769675c48-rn4w6\" (UID: \"03a1c734-f7c7-45d2-9868-eb943d7f28c9\") " pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:39.434714 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.434628    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:39.594921 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.594875    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb" event={"ID":"bec8c6aa-d55b-4505-9f46-740144961193","Type":"ContainerStarted","Data":"8e3689304da4bbd683555218168169f783f68dd5e6d206c668154aedb7f49f0b"}
Sep 06 19:33:39.706459 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:39.706205    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"]
Sep 06 19:33:39.709278 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:33:39.709245    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a1c734_f7c7_45d2_9868_eb943d7f28c9.slice/crio-13806463ebe66a52bc396282edc172ffd08dbde1aa13ca4da18902420370c0e9 WatchSource:0}: Error finding container 13806463ebe66a52bc396282edc172ffd08dbde1aa13ca4da18902420370c0e9: Status 404 returned error can't find the container with id 13806463ebe66a52bc396282edc172ffd08dbde1aa13ca4da18902420370c0e9
Sep 06 19:33:40.602235 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:40.602187    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6" event={"ID":"03a1c734-f7c7-45d2-9868-eb943d7f28c9","Type":"ContainerStarted","Data":"13806463ebe66a52bc396282edc172ffd08dbde1aa13ca4da18902420370c0e9"}
Sep 06 19:33:42.614038 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:42.613997    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb" event={"ID":"bec8c6aa-d55b-4505-9f46-740144961193","Type":"ContainerStarted","Data":"6f714ea8ebc325db0b61a018f80760bf31aad07d9a2523b114040d4f4c93e71f"}
Sep 06 19:33:44.625633 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:44.625595    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6" event={"ID":"03a1c734-f7c7-45d2-9868-eb943d7f28c9","Type":"ContainerStarted","Data":"b09ce06b3a0cf7fb5154fa50650a8eac85fdf8a606d3cfd44d625b131a737346"}
Sep 06 19:33:44.626188 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:44.625691    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:44.627195 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:44.627173    2571 generic.go:356] "Generic (PLEG): container finished" podID="bec8c6aa-d55b-4505-9f46-740144961193" containerID="6f714ea8ebc325db0b61a018f80760bf31aad07d9a2523b114040d4f4c93e71f" exitCode=0
Sep 06 19:33:44.627288 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:44.627221    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb" event={"ID":"bec8c6aa-d55b-4505-9f46-740144961193","Type":"ContainerDied","Data":"6f714ea8ebc325db0b61a018f80760bf31aad07d9a2523b114040d4f4c93e71f"}
Sep 06 19:33:44.640887 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:44.640839    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6" podStartSLOduration=1.591030196 podStartE2EDuration="5.640822882s" totalImagesPullingTime="4.049792686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:33:44.639772835 +0000 UTC m=+388.210071182" watchObservedRunningTime="2026-09-06 19:33:44.640822882 +0000 UTC m=+388.211121285"
Sep 06 19:33:45.633394 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:45.633360    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb" event={"ID":"bec8c6aa-d55b-4505-9f46-740144961193","Type":"ContainerStarted","Data":"4b93c0a2ce73339ff6861d1e8b3e286a9d386f9b6962d332d3be6e8843651cf8"}
Sep 06 19:33:45.633783 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:45.633630    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:33:45.653718 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:45.653664    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb" podStartSLOduration=3.178056169 podStartE2EDuration="7.653646729s" totalImagesPullingTime="3.149733805s" totalInitContainerRuntime="1.325856755s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:33:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:33:45.652368082 +0000 UTC m=+389.222666464" watchObservedRunningTime="2026-09-06 19:33:45.653646729 +0000 UTC m=+389.223945075"
Sep 06 19:33:55.636066 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:55.636031    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7769675c48-rn4w6"
Sep 06 19:33:56.640198 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:33:56.640163    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f964dcc99-npwhb"
Sep 06 19:34:03.655189 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.655152    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv"]
Sep 06 19:34:03.668262 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.668224    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv"
Sep 06 19:34:03.669074 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.669046    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv"]
Sep 06 19:34:03.670937 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.670913    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 06 19:34:03.670937 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.670930    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 06 19:34:03.671161 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.671142    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 06 19:34:03.671789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.671763    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 06 19:34:03.671927 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.671910    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9wnfx\""
Sep 06 19:34:03.710703 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.710668    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstl7\" (UniqueName: \"kubernetes.io/projected/96e685d1-d3e3-43b8-b79b-88159bde0379-kube-api-access-xstl7\") pod \"kube-auth-proxy-7d55646b9-qk9vv\" (UID: \"96e685d1-d3e3-43b8-b79b-88159bde0379\") " pod="openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv"
Sep 06 19:34:03.710891 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.710729    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96e685d1-d3e3-43b8-b79b-88159bde0379-tmp\") pod \"kube-auth-proxy-7d55646b9-qk9vv\" (UID: \"96e685d1-d3e3-43b8-b79b-88159bde0379\") " pod="openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv"
Sep 06 19:34:03.710891 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.710817    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e685d1-d3e3-43b8-b79b-88159bde0379-tls-certs\") pod \"kube-auth-proxy-7d55646b9-qk9vv\" (UID: \"96e685d1-d3e3-43b8-b79b-88159bde0379\") " pod="openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv"
Sep 06 19:34:03.811649 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.811602    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e685d1-d3e3-43b8-b79b-88159bde0379-tls-certs\") pod \"kube-auth-proxy-7d55646b9-qk9vv\" (UID: \"96e685d1-d3e3-43b8-b79b-88159bde0379\") " pod="openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv"
Sep 06 19:34:03.811852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.811671    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstl7\" (UniqueName: \"kubernetes.io/projected/96e685d1-d3e3-43b8-b79b-88159bde0379-kube-api-access-xstl7\") pod \"kube-auth-proxy-7d55646b9-qk9vv\" (UID: \"96e685d1-d3e3-43b8-b79b-88159bde0379\") " pod="openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv"
Sep 06 19:34:03.811852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.811722    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96e685d1-d3e3-43b8-b79b-88159bde0379-tmp\") pod \"kube-auth-proxy-7d55646b9-qk9vv\" (UID: \"96e685d1-d3e3-43b8-b79b-88159bde0379\") " pod="openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv"
Sep 06 19:34:03.811852 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:34:03.811766    2571 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 06 19:34:03.812060 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:34:03.811862    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e685d1-d3e3-43b8-b79b-88159bde0379-tls-certs podName:96e685d1-d3e3-43b8-b79b-88159bde0379 nodeName:}" failed. No retries permitted until 2026-09-06 19:34:04.311840314 +0000 UTC m=+407.882138654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/96e685d1-d3e3-43b8-b79b-88159bde0379-tls-certs") pod "kube-auth-proxy-7d55646b9-qk9vv" (UID: "96e685d1-d3e3-43b8-b79b-88159bde0379") : secret "kube-auth-proxy-tls" not found
Sep 06 19:34:03.814035 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.814010    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96e685d1-d3e3-43b8-b79b-88159bde0379-tmp\") pod \"kube-auth-proxy-7d55646b9-qk9vv\" (UID: \"96e685d1-d3e3-43b8-b79b-88159bde0379\") " pod="openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv"
Sep 06 19:34:03.819606 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:03.819581    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstl7\" (UniqueName: \"kubernetes.io/projected/96e685d1-d3e3-43b8-b79b-88159bde0379-kube-api-access-xstl7\") pod \"kube-auth-proxy-7d55646b9-qk9vv\" (UID: \"96e685d1-d3e3-43b8-b79b-88159bde0379\") " pod="openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv"
Sep 06 19:34:04.315790 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:04.315748    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e685d1-d3e3-43b8-b79b-88159bde0379-tls-certs\") pod \"kube-auth-proxy-7d55646b9-qk9vv\" (UID: \"96e685d1-d3e3-43b8-b79b-88159bde0379\") " pod="openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv"
Sep 06 19:34:04.318259 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:04.318232    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e685d1-d3e3-43b8-b79b-88159bde0379-tls-certs\") pod \"kube-auth-proxy-7d55646b9-qk9vv\" (UID: \"96e685d1-d3e3-43b8-b79b-88159bde0379\") " pod="openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv"
Sep 06 19:34:04.580338 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:04.580240    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv"
Sep 06 19:34:04.849570 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:04.849545    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv"]
Sep 06 19:34:04.852007 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:34:04.851979    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e685d1_d3e3_43b8_b79b_88159bde0379.slice/crio-cebeb85d415790c25061e3a432c65caabdb7d226142dd510df409547a8af39b5 WatchSource:0}: Error finding container cebeb85d415790c25061e3a432c65caabdb7d226142dd510df409547a8af39b5: Status 404 returned error can't find the container with id cebeb85d415790c25061e3a432c65caabdb7d226142dd510df409547a8af39b5
Sep 06 19:34:05.726585 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:05.726541    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv" event={"ID":"96e685d1-d3e3-43b8-b79b-88159bde0379","Type":"ContainerStarted","Data":"cebeb85d415790c25061e3a432c65caabdb7d226142dd510df409547a8af39b5"}
Sep 06 19:34:07.738840 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:07.738743    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv" event={"ID":"96e685d1-d3e3-43b8-b79b-88159bde0379","Type":"ContainerStarted","Data":"9215256c18dc95569655f4a695eb3907276769508d6497f18a561159c67698d5"}
Sep 06 19:34:07.754088 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:34:07.754030    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d55646b9-qk9vv" podStartSLOduration=2.160533961 podStartE2EDuration="4.75401234s" totalImagesPullingTime="2.593478379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:34:07.752544898 +0000 UTC m=+411.322843247" watchObservedRunningTime="2026-09-06 19:34:07.75401234 +0000 UTC m=+411.324310686"
Sep 06 19:35:46.534479 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:46.534440    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"]
Sep 06 19:35:46.537885 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:46.537867    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"
Sep 06 19:35:46.541017 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:46.540995    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 06 19:35:46.541258 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:46.541238    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-986z8\""
Sep 06 19:35:46.541307 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:46.541280    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 06 19:35:46.555233 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:46.555206    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"]
Sep 06 19:35:46.606264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:46.606229    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb59d\" (UniqueName: \"kubernetes.io/projected/435e2c41-96cc-4ef7-a3e7-ff4fd0f85739-kube-api-access-lb59d\") pod \"kuadrant-operator-controller-manager-6bd956ccf-nhcf4\" (UID: \"435e2c41-96cc-4ef7-a3e7-ff4fd0f85739\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"
Sep 06 19:35:46.606435 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:46.606336    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/435e2c41-96cc-4ef7-a3e7-ff4fd0f85739-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-nhcf4\" (UID: \"435e2c41-96cc-4ef7-a3e7-ff4fd0f85739\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"
Sep 06 19:35:46.707561 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:46.707518    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/435e2c41-96cc-4ef7-a3e7-ff4fd0f85739-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-nhcf4\" (UID: \"435e2c41-96cc-4ef7-a3e7-ff4fd0f85739\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"
Sep 06 19:35:46.707732 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:46.707582    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb59d\" (UniqueName: \"kubernetes.io/projected/435e2c41-96cc-4ef7-a3e7-ff4fd0f85739-kube-api-access-lb59d\") pod \"kuadrant-operator-controller-manager-6bd956ccf-nhcf4\" (UID: \"435e2c41-96cc-4ef7-a3e7-ff4fd0f85739\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"
Sep 06 19:35:46.707996 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:46.707947    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/435e2c41-96cc-4ef7-a3e7-ff4fd0f85739-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-nhcf4\" (UID: \"435e2c41-96cc-4ef7-a3e7-ff4fd0f85739\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"
Sep 06 19:35:46.718973 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:46.718935    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb59d\" (UniqueName: \"kubernetes.io/projected/435e2c41-96cc-4ef7-a3e7-ff4fd0f85739-kube-api-access-lb59d\") pod \"kuadrant-operator-controller-manager-6bd956ccf-nhcf4\" (UID: \"435e2c41-96cc-4ef7-a3e7-ff4fd0f85739\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"
Sep 06 19:35:46.848219 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:46.848184    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"
Sep 06 19:35:47.133074 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:47.132979    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"]
Sep 06 19:35:47.135405 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:35:47.135373    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435e2c41_96cc_4ef7_a3e7_ff4fd0f85739.slice/crio-2bc2e778c84e5fd92894a43b5fbac61ee156c11060606b4b763b737f7d55faf8 WatchSource:0}: Error finding container 2bc2e778c84e5fd92894a43b5fbac61ee156c11060606b4b763b737f7d55faf8: Status 404 returned error can't find the container with id 2bc2e778c84e5fd92894a43b5fbac61ee156c11060606b4b763b737f7d55faf8
Sep 06 19:35:47.144761 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:47.144733    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"]
Sep 06 19:35:47.153045 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:47.153020    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"]
Sep 06 19:35:47.171944 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:47.171910    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"]
Sep 06 19:35:47.177254 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:47.177227    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"
Sep 06 19:35:47.189278 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:47.189238    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"]
Sep 06 19:35:47.212039 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:47.212002    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6b23484-e18f-4995-a7dd-b45b10c1fee3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-5cqx7\" (UID: \"e6b23484-e18f-4995-a7dd-b45b10c1fee3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"
Sep 06 19:35:47.212220 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:47.212063    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqt84\" (UniqueName: \"kubernetes.io/projected/e6b23484-e18f-4995-a7dd-b45b10c1fee3-kube-api-access-wqt84\") pod \"kuadrant-operator-controller-manager-6bd956ccf-5cqx7\" (UID: \"e6b23484-e18f-4995-a7dd-b45b10c1fee3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"
Sep 06 19:35:47.313495 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:47.313451    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6b23484-e18f-4995-a7dd-b45b10c1fee3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-5cqx7\" (UID: \"e6b23484-e18f-4995-a7dd-b45b10c1fee3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"
Sep 06 19:35:47.313695 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:47.313508    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqt84\" (UniqueName: \"kubernetes.io/projected/e6b23484-e18f-4995-a7dd-b45b10c1fee3-kube-api-access-wqt84\") pod \"kuadrant-operator-controller-manager-6bd956ccf-5cqx7\" (UID: \"e6b23484-e18f-4995-a7dd-b45b10c1fee3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"
Sep 06 19:35:47.313894 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:47.313872    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6b23484-e18f-4995-a7dd-b45b10c1fee3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-5cqx7\" (UID: \"e6b23484-e18f-4995-a7dd-b45b10c1fee3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"
Sep 06 19:35:47.334013 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:47.333978    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqt84\" (UniqueName: \"kubernetes.io/projected/e6b23484-e18f-4995-a7dd-b45b10c1fee3-kube-api-access-wqt84\") pod \"kuadrant-operator-controller-manager-6bd956ccf-5cqx7\" (UID: \"e6b23484-e18f-4995-a7dd-b45b10c1fee3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"
Sep 06 19:35:47.488887 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:47.488799    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"
Sep 06 19:35:47.767539 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:47.767513    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"]
Sep 06 19:35:47.770202 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:35:47.770171    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b23484_e18f_4995_a7dd_b45b10c1fee3.slice/crio-ca90aa8197f2e89fb87cf4f3240f1224d8cf9b28df1275926fcd322b44aa68f5 WatchSource:0}: Error finding container ca90aa8197f2e89fb87cf4f3240f1224d8cf9b28df1275926fcd322b44aa68f5: Status 404 returned error can't find the container with id ca90aa8197f2e89fb87cf4f3240f1224d8cf9b28df1275926fcd322b44aa68f5
Sep 06 19:35:48.192931 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:48.192889    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7" event={"ID":"e6b23484-e18f-4995-a7dd-b45b10c1fee3","Type":"ContainerStarted","Data":"ca90aa8197f2e89fb87cf4f3240f1224d8cf9b28df1275926fcd322b44aa68f5"}
Sep 06 19:35:52.217231 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:52.217190    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7" event={"ID":"e6b23484-e18f-4995-a7dd-b45b10c1fee3","Type":"ContainerStarted","Data":"b1b3a627abe7c267ad60b53ef748ed8d42ff0c9f8bfc719d66a9878098987459"}
Sep 06 19:35:52.217623 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:52.217264    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"
Sep 06 19:35:52.219022 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:52.218994    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4" podUID="435e2c41-96cc-4ef7-a3e7-ff4fd0f85739" containerName="manager" containerID="cri-o://381ebebb95d2299ac03a7a3ec53cef34fff6c47475983f10068affe33a5604e7" gracePeriod=2
Sep 06 19:35:52.236739 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:52.236672    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7" podStartSLOduration=1.59784755 podStartE2EDuration="5.236657949s" totalImagesPullingTime="3.638810399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:35:52.235539364 +0000 UTC m=+515.805837713" watchObservedRunningTime="2026-09-06 19:35:52.236657949 +0000 UTC m=+515.806956294"
Sep 06 19:35:52.463504 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:52.463474    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"
Sep 06 19:35:52.465426 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:35:52.465402    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-nhcf4\" is forbidden: User \"system:node:ip-10-0-141-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-42.ec2.internal' and this object" podUID="435e2c41-96cc-4ef7-a3e7-ff4fd0f85739" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"
Sep 06 19:35:52.563617 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:52.563592    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/435e2c41-96cc-4ef7-a3e7-ff4fd0f85739-extensions-socket-volume\") pod \"435e2c41-96cc-4ef7-a3e7-ff4fd0f85739\" (UID: \"435e2c41-96cc-4ef7-a3e7-ff4fd0f85739\") "
Sep 06 19:35:52.563767 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:52.563698    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb59d\" (UniqueName: \"kubernetes.io/projected/435e2c41-96cc-4ef7-a3e7-ff4fd0f85739-kube-api-access-lb59d\") pod \"435e2c41-96cc-4ef7-a3e7-ff4fd0f85739\" (UID: \"435e2c41-96cc-4ef7-a3e7-ff4fd0f85739\") "
Sep 06 19:35:52.563915 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:52.563894    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435e2c41-96cc-4ef7-a3e7-ff4fd0f85739-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "435e2c41-96cc-4ef7-a3e7-ff4fd0f85739" (UID: "435e2c41-96cc-4ef7-a3e7-ff4fd0f85739"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:35:52.564003 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:52.563988    2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/435e2c41-96cc-4ef7-a3e7-ff4fd0f85739-extensions-socket-volume\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:35:52.565851 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:52.565828    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435e2c41-96cc-4ef7-a3e7-ff4fd0f85739-kube-api-access-lb59d" (OuterVolumeSpecName: "kube-api-access-lb59d") pod "435e2c41-96cc-4ef7-a3e7-ff4fd0f85739" (UID: "435e2c41-96cc-4ef7-a3e7-ff4fd0f85739"). InnerVolumeSpecName "kube-api-access-lb59d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:35:52.665163 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:52.665124    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb59d\" (UniqueName: \"kubernetes.io/projected/435e2c41-96cc-4ef7-a3e7-ff4fd0f85739-kube-api-access-lb59d\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:35:52.918494 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:52.918411    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="435e2c41-96cc-4ef7-a3e7-ff4fd0f85739" path="/var/lib/kubelet/pods/435e2c41-96cc-4ef7-a3e7-ff4fd0f85739/volumes"
Sep 06 19:35:53.225190 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:53.225104    2571 generic.go:356] "Generic (PLEG): container finished" podID="435e2c41-96cc-4ef7-a3e7-ff4fd0f85739" containerID="381ebebb95d2299ac03a7a3ec53cef34fff6c47475983f10068affe33a5604e7" exitCode=2
Sep 06 19:35:53.225190 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:53.225149    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"
Sep 06 19:35:53.225654 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:53.225210    2571 scope.go:117] "RemoveContainer" containerID="381ebebb95d2299ac03a7a3ec53cef34fff6c47475983f10068affe33a5604e7"
Sep 06 19:35:53.229540 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:35:53.229507    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-nhcf4\" is forbidden: User \"system:node:ip-10-0-141-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-42.ec2.internal' and this object" podUID="435e2c41-96cc-4ef7-a3e7-ff4fd0f85739" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-nhcf4"
Sep 06 19:35:53.233838 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:53.233818    2571 scope.go:117] "RemoveContainer" containerID="381ebebb95d2299ac03a7a3ec53cef34fff6c47475983f10068affe33a5604e7"
Sep 06 19:35:53.234111 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:35:53.234091    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"381ebebb95d2299ac03a7a3ec53cef34fff6c47475983f10068affe33a5604e7\": container with ID starting with 381ebebb95d2299ac03a7a3ec53cef34fff6c47475983f10068affe33a5604e7 not found: ID does not exist" containerID="381ebebb95d2299ac03a7a3ec53cef34fff6c47475983f10068affe33a5604e7"
Sep 06 19:35:53.234168 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:35:53.234120    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"381ebebb95d2299ac03a7a3ec53cef34fff6c47475983f10068affe33a5604e7"} err="failed to get container status \"381ebebb95d2299ac03a7a3ec53cef34fff6c47475983f10068affe33a5604e7\": rpc error: code = NotFound desc = could not find container \"381ebebb95d2299ac03a7a3ec53cef34fff6c47475983f10068affe33a5604e7\": container with ID starting with 381ebebb95d2299ac03a7a3ec53cef34fff6c47475983f10068affe33a5604e7 not found: ID does not exist"
Sep 06 19:36:03.228031 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:03.227999    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"
Sep 06 19:36:16.282442 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:16.282364    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"]
Sep 06 19:36:16.282855 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:16.282561    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7" podUID="e6b23484-e18f-4995-a7dd-b45b10c1fee3" containerName="manager" containerID="cri-o://b1b3a627abe7c267ad60b53ef748ed8d42ff0c9f8bfc719d66a9878098987459" gracePeriod=10
Sep 06 19:36:16.529596 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:16.529566    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"
Sep 06 19:36:16.683256 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:16.683219    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqt84\" (UniqueName: \"kubernetes.io/projected/e6b23484-e18f-4995-a7dd-b45b10c1fee3-kube-api-access-wqt84\") pod \"e6b23484-e18f-4995-a7dd-b45b10c1fee3\" (UID: \"e6b23484-e18f-4995-a7dd-b45b10c1fee3\") "
Sep 06 19:36:16.683457 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:16.683276    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6b23484-e18f-4995-a7dd-b45b10c1fee3-extensions-socket-volume\") pod \"e6b23484-e18f-4995-a7dd-b45b10c1fee3\" (UID: \"e6b23484-e18f-4995-a7dd-b45b10c1fee3\") "
Sep 06 19:36:16.683785 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:16.683748    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b23484-e18f-4995-a7dd-b45b10c1fee3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e6b23484-e18f-4995-a7dd-b45b10c1fee3" (UID: "e6b23484-e18f-4995-a7dd-b45b10c1fee3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:36:16.685382 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:16.685361    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b23484-e18f-4995-a7dd-b45b10c1fee3-kube-api-access-wqt84" (OuterVolumeSpecName: "kube-api-access-wqt84") pod "e6b23484-e18f-4995-a7dd-b45b10c1fee3" (UID: "e6b23484-e18f-4995-a7dd-b45b10c1fee3"). InnerVolumeSpecName "kube-api-access-wqt84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:36:16.784722 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:16.784683    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqt84\" (UniqueName: \"kubernetes.io/projected/e6b23484-e18f-4995-a7dd-b45b10c1fee3-kube-api-access-wqt84\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:36:16.784722 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:16.784714    2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6b23484-e18f-4995-a7dd-b45b10c1fee3-extensions-socket-volume\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:36:17.343592 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:17.343556    2571 generic.go:356] "Generic (PLEG): container finished" podID="e6b23484-e18f-4995-a7dd-b45b10c1fee3" containerID="b1b3a627abe7c267ad60b53ef748ed8d42ff0c9f8bfc719d66a9878098987459" exitCode=0
Sep 06 19:36:17.344023 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:17.343607    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"
Sep 06 19:36:17.344023 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:17.343641    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7" event={"ID":"e6b23484-e18f-4995-a7dd-b45b10c1fee3","Type":"ContainerDied","Data":"b1b3a627abe7c267ad60b53ef748ed8d42ff0c9f8bfc719d66a9878098987459"}
Sep 06 19:36:17.344023 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:17.343697    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7" event={"ID":"e6b23484-e18f-4995-a7dd-b45b10c1fee3","Type":"ContainerDied","Data":"ca90aa8197f2e89fb87cf4f3240f1224d8cf9b28df1275926fcd322b44aa68f5"}
Sep 06 19:36:17.344023 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:17.343720    2571 scope.go:117] "RemoveContainer" containerID="b1b3a627abe7c267ad60b53ef748ed8d42ff0c9f8bfc719d66a9878098987459"
Sep 06 19:36:17.352705 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:17.352675    2571 scope.go:117] "RemoveContainer" containerID="b1b3a627abe7c267ad60b53ef748ed8d42ff0c9f8bfc719d66a9878098987459"
Sep 06 19:36:17.352980 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:36:17.352947    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b3a627abe7c267ad60b53ef748ed8d42ff0c9f8bfc719d66a9878098987459\": container with ID starting with b1b3a627abe7c267ad60b53ef748ed8d42ff0c9f8bfc719d66a9878098987459 not found: ID does not exist" containerID="b1b3a627abe7c267ad60b53ef748ed8d42ff0c9f8bfc719d66a9878098987459"
Sep 06 19:36:17.353064 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:17.352992    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b3a627abe7c267ad60b53ef748ed8d42ff0c9f8bfc719d66a9878098987459"} err="failed to get container status \"b1b3a627abe7c267ad60b53ef748ed8d42ff0c9f8bfc719d66a9878098987459\": rpc error: code = NotFound desc = could not find container \"b1b3a627abe7c267ad60b53ef748ed8d42ff0c9f8bfc719d66a9878098987459\": container with ID starting with b1b3a627abe7c267ad60b53ef748ed8d42ff0c9f8bfc719d66a9878098987459 not found: ID does not exist"
Sep 06 19:36:17.364201 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:17.364169    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"]
Sep 06 19:36:17.378116 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:17.378091    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-5cqx7"]
Sep 06 19:36:18.917832 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:18.917788    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6b23484-e18f-4995-a7dd-b45b10c1fee3" path="/var/lib/kubelet/pods/e6b23484-e18f-4995-a7dd-b45b10c1fee3/volumes"
Sep 06 19:36:19.536358 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.536317    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"]
Sep 06 19:36:19.536791 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.536776    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="435e2c41-96cc-4ef7-a3e7-ff4fd0f85739" containerName="manager"
Sep 06 19:36:19.536882 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.536793    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="435e2c41-96cc-4ef7-a3e7-ff4fd0f85739" containerName="manager"
Sep 06 19:36:19.536882 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.536824    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b23484-e18f-4995-a7dd-b45b10c1fee3" containerName="manager"
Sep 06 19:36:19.536882 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.536832    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b23484-e18f-4995-a7dd-b45b10c1fee3" containerName="manager"
Sep 06 19:36:19.537055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.536918    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6b23484-e18f-4995-a7dd-b45b10c1fee3" containerName="manager"
Sep 06 19:36:19.537055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.536928    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="435e2c41-96cc-4ef7-a3e7-ff4fd0f85739" containerName="manager"
Sep 06 19:36:19.541428 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.541403    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.543856 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.543833    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 06 19:36:19.544000 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.543976    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c8n9m\""
Sep 06 19:36:19.551119 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.551093    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"]
Sep 06 19:36:19.710459 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.710421    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c1898a-3846-4250-8601-fbc9bb7fcc70-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.710459 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.710459    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c1898a-3846-4250-8601-fbc9bb7fcc70-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.710674 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.710491    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c1898a-3846-4250-8601-fbc9bb7fcc70-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.710674 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.710512    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8zjs\" (UniqueName: \"kubernetes.io/projected/c7c1898a-3846-4250-8601-fbc9bb7fcc70-kube-api-access-z8zjs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.710674 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.710556    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7c1898a-3846-4250-8601-fbc9bb7fcc70-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.710674 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.710594    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c1898a-3846-4250-8601-fbc9bb7fcc70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.710674 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.710617    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c1898a-3846-4250-8601-fbc9bb7fcc70-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.710674 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.710658    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c1898a-3846-4250-8601-fbc9bb7fcc70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.710872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.710722    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c1898a-3846-4250-8601-fbc9bb7fcc70-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.811921 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.811890    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7c1898a-3846-4250-8601-fbc9bb7fcc70-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.812046 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.811936    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c1898a-3846-4250-8601-fbc9bb7fcc70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.812046 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.811988    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c1898a-3846-4250-8601-fbc9bb7fcc70-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.812046 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.812011    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c1898a-3846-4250-8601-fbc9bb7fcc70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.812046 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.812030    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c1898a-3846-4250-8601-fbc9bb7fcc70-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.812218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.812072    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c1898a-3846-4250-8601-fbc9bb7fcc70-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.812218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.812093    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c1898a-3846-4250-8601-fbc9bb7fcc70-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.812218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.812120    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c1898a-3846-4250-8601-fbc9bb7fcc70-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.812218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.812149    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8zjs\" (UniqueName: \"kubernetes.io/projected/c7c1898a-3846-4250-8601-fbc9bb7fcc70-kube-api-access-z8zjs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.812453 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.812428    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7c1898a-3846-4250-8601-fbc9bb7fcc70-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.812518 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.812483    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7c1898a-3846-4250-8601-fbc9bb7fcc70-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.812575 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.812523    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c1898a-3846-4250-8601-fbc9bb7fcc70-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.812631 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.812571    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7c1898a-3846-4250-8601-fbc9bb7fcc70-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.812865 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.812842    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7c1898a-3846-4250-8601-fbc9bb7fcc70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.814534 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.814507    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7c1898a-3846-4250-8601-fbc9bb7fcc70-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.814634 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.814570    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7c1898a-3846-4250-8601-fbc9bb7fcc70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.819347 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.819315    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8zjs\" (UniqueName: \"kubernetes.io/projected/c7c1898a-3846-4250-8601-fbc9bb7fcc70-kube-api-access-z8zjs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.819441 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.819345    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7c1898a-3846-4250-8601-fbc9bb7fcc70-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-95l97\" (UID: \"c7c1898a-3846-4250-8601-fbc9bb7fcc70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:19.852477 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:19.852443    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:20.130351 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:20.130318    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"]
Sep 06 19:36:20.131941 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:36:20.131920    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c1898a_3846_4250_8601_fbc9bb7fcc70.slice/crio-95ec67c9c7b9ba91d940b1a6e201437e9c22b8db85f82abea198e25896969c61 WatchSource:0}: Error finding container 95ec67c9c7b9ba91d940b1a6e201437e9c22b8db85f82abea198e25896969c61: Status 404 returned error can't find the container with id 95ec67c9c7b9ba91d940b1a6e201437e9c22b8db85f82abea198e25896969c61
Sep 06 19:36:20.365567 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:20.365526    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97" event={"ID":"c7c1898a-3846-4250-8601-fbc9bb7fcc70","Type":"ContainerStarted","Data":"95ec67c9c7b9ba91d940b1a6e201437e9c22b8db85f82abea198e25896969c61"}
Sep 06 19:36:23.442931 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:23.442885    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:36:23.443193 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:23.443001    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:36:23.443193 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:23.443038    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:36:24.389097 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:24.389054    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97" event={"ID":"c7c1898a-3846-4250-8601-fbc9bb7fcc70","Type":"ContainerStarted","Data":"9f47d0bc9e3eeb8c871a09f38f008125f202867a43aa514c16b40f4dcb95843c"}
Sep 06 19:36:24.409363 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:24.409304    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97" podStartSLOduration=2.101130769 podStartE2EDuration="5.409288901s" totalImagesPullingTime="3.308158132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:36:24.408113074 +0000 UTC m=+547.978411418" watchObservedRunningTime="2026-09-06 19:36:24.409288901 +0000 UTC m=+547.979587247"
Sep 06 19:36:24.853230 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:24.853190    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:24.858099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:24.858072    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:25.393722 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:25.393694    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:25.394746 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:25.394724    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:36:34.311452 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.311414    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:36:34.313834 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.313816    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"
Sep 06 19:36:34.315947 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.315921    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 06 19:36:34.316089 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.316064    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zq5kt\""
Sep 06 19:36:34.316925 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.316903    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 06 19:36:34.317044 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.316908    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 06 19:36:34.321807 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.321778    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:36:34.344210 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.344169    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d41b9aee-d19d-4b66-82c3-5be20b9c9718-config-file\") pod \"limitador-limitador-85d9db4d5d-f5f2g\" (UID: \"d41b9aee-d19d-4b66-82c3-5be20b9c9718\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"
Sep 06 19:36:34.344408 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.344341    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7kg\" (UniqueName: \"kubernetes.io/projected/d41b9aee-d19d-4b66-82c3-5be20b9c9718-kube-api-access-qn7kg\") pod \"limitador-limitador-85d9db4d5d-f5f2g\" (UID: \"d41b9aee-d19d-4b66-82c3-5be20b9c9718\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"
Sep 06 19:36:34.347207 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.347179    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:36:34.445092 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.445056    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d41b9aee-d19d-4b66-82c3-5be20b9c9718-config-file\") pod \"limitador-limitador-85d9db4d5d-f5f2g\" (UID: \"d41b9aee-d19d-4b66-82c3-5be20b9c9718\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"
Sep 06 19:36:34.445287 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.445148    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7kg\" (UniqueName: \"kubernetes.io/projected/d41b9aee-d19d-4b66-82c3-5be20b9c9718-kube-api-access-qn7kg\") pod \"limitador-limitador-85d9db4d5d-f5f2g\" (UID: \"d41b9aee-d19d-4b66-82c3-5be20b9c9718\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"
Sep 06 19:36:34.445741 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.445715    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d41b9aee-d19d-4b66-82c3-5be20b9c9718-config-file\") pod \"limitador-limitador-85d9db4d5d-f5f2g\" (UID: \"d41b9aee-d19d-4b66-82c3-5be20b9c9718\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"
Sep 06 19:36:34.452634 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.452602    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7kg\" (UniqueName: \"kubernetes.io/projected/d41b9aee-d19d-4b66-82c3-5be20b9c9718-kube-api-access-qn7kg\") pod \"limitador-limitador-85d9db4d5d-f5f2g\" (UID: \"d41b9aee-d19d-4b66-82c3-5be20b9c9718\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"
Sep 06 19:36:34.624793 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.624703    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"
Sep 06 19:36:34.895770 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.895719    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:36:34.898082 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:36:34.898057    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41b9aee_d19d_4b66_82c3_5be20b9c9718.slice/crio-b0b82d6249e72a73dc71d61d650cc0d02bce315bc1487ae4b3d2435c42fd1eeb WatchSource:0}: Error finding container b0b82d6249e72a73dc71d61d650cc0d02bce315bc1487ae4b3d2435c42fd1eeb: Status 404 returned error can't find the container with id b0b82d6249e72a73dc71d61d650cc0d02bce315bc1487ae4b3d2435c42fd1eeb
Sep 06 19:36:34.918101 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.918070    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-hnsbn"]
Sep 06 19:36:34.921495 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.921467    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-hnsbn"
Sep 06 19:36:34.923224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.923201    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-hnsbn"]
Sep 06 19:36:34.923600 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.923578    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kmgmx\""
Sep 06 19:36:34.950144 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:34.950113    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mk2r\" (UniqueName: \"kubernetes.io/projected/0b9625a5-9fbb-4d84-91de-958118635e9a-kube-api-access-9mk2r\") pod \"authorino-c548b5d7d-hnsbn\" (UID: \"0b9625a5-9fbb-4d84-91de-958118635e9a\") " pod="kuadrant-system/authorino-c548b5d7d-hnsbn"
Sep 06 19:36:35.050930 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:35.050894    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mk2r\" (UniqueName: \"kubernetes.io/projected/0b9625a5-9fbb-4d84-91de-958118635e9a-kube-api-access-9mk2r\") pod \"authorino-c548b5d7d-hnsbn\" (UID: \"0b9625a5-9fbb-4d84-91de-958118635e9a\") " pod="kuadrant-system/authorino-c548b5d7d-hnsbn"
Sep 06 19:36:35.058324 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:35.058296    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mk2r\" (UniqueName: \"kubernetes.io/projected/0b9625a5-9fbb-4d84-91de-958118635e9a-kube-api-access-9mk2r\") pod \"authorino-c548b5d7d-hnsbn\" (UID: \"0b9625a5-9fbb-4d84-91de-958118635e9a\") " pod="kuadrant-system/authorino-c548b5d7d-hnsbn"
Sep 06 19:36:35.231824 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:35.231715    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-hnsbn"
Sep 06 19:36:35.442747 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:35.442707    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g" event={"ID":"d41b9aee-d19d-4b66-82c3-5be20b9c9718","Type":"ContainerStarted","Data":"b0b82d6249e72a73dc71d61d650cc0d02bce315bc1487ae4b3d2435c42fd1eeb"}
Sep 06 19:36:35.499431 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:35.499404    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-hnsbn"]
Sep 06 19:36:35.502074 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:36:35.502045    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b9625a5_9fbb_4d84_91de_958118635e9a.slice/crio-f38c839cbbf517faa30957e2f32c53c3e21b4f9fcec90203c0e8e41e8c2507c6 WatchSource:0}: Error finding container f38c839cbbf517faa30957e2f32c53c3e21b4f9fcec90203c0e8e41e8c2507c6: Status 404 returned error can't find the container with id f38c839cbbf517faa30957e2f32c53c3e21b4f9fcec90203c0e8e41e8c2507c6
Sep 06 19:36:36.450157 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:36.450093    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-hnsbn" event={"ID":"0b9625a5-9fbb-4d84-91de-958118635e9a","Type":"ContainerStarted","Data":"f38c839cbbf517faa30957e2f32c53c3e21b4f9fcec90203c0e8e41e8c2507c6"}
Sep 06 19:36:36.452342 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:36.452302    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g" event={"ID":"d41b9aee-d19d-4b66-82c3-5be20b9c9718","Type":"ContainerStarted","Data":"f35f520f54496773acd903e1bab16c1665cacff569be68b289cafd1b5a4634be"}
Sep 06 19:36:36.452610 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:36.452560    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"
Sep 06 19:36:36.468152 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:36.468090    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g" podStartSLOduration=1.146503716 podStartE2EDuration="2.468069614s" totalImagesPullingTime="1.321565898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:36:36.467042949 +0000 UTC m=+560.037341296" watchObservedRunningTime="2026-09-06 19:36:36.468069614 +0000 UTC m=+560.038367961"
Sep 06 19:36:37.459844 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:37.459808    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-hnsbn" event={"ID":"0b9625a5-9fbb-4d84-91de-958118635e9a","Type":"ContainerStarted","Data":"8986a68625ccb47f28e29e96383b1ca2bd8b5dad3922315c8aca762947cd3984"}
Sep 06 19:36:37.474132 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:37.474083    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-hnsbn" podStartSLOduration=1.6462320350000001 podStartE2EDuration="3.47406711s" totalImagesPullingTime="1.827835075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:36:37.472730934 +0000 UTC m=+561.043029280" watchObservedRunningTime="2026-09-06 19:36:37.47406711 +0000 UTC m=+561.044365456"
Sep 06 19:36:47.461495 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:36:47.461460    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"
Sep 06 19:37:04.298769 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:04.298730    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64b5794f6c-9cvtw"]
Sep 06 19:37:04.308146 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:04.308120    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64b5794f6c-9cvtw"]
Sep 06 19:37:04.308304 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:04.308223    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b5794f6c-9cvtw"
Sep 06 19:37:04.310504 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:04.310481    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 06 19:37:04.408916 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:04.408870    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c85436c5-ec45-4464-bf21-418ee202897a-tls-cert\") pod \"authorino-64b5794f6c-9cvtw\" (UID: \"c85436c5-ec45-4464-bf21-418ee202897a\") " pod="kuadrant-system/authorino-64b5794f6c-9cvtw"
Sep 06 19:37:04.408916 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:04.408925    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n272d\" (UniqueName: \"kubernetes.io/projected/c85436c5-ec45-4464-bf21-418ee202897a-kube-api-access-n272d\") pod \"authorino-64b5794f6c-9cvtw\" (UID: \"c85436c5-ec45-4464-bf21-418ee202897a\") " pod="kuadrant-system/authorino-64b5794f6c-9cvtw"
Sep 06 19:37:04.509747 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:04.509707    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c85436c5-ec45-4464-bf21-418ee202897a-tls-cert\") pod \"authorino-64b5794f6c-9cvtw\" (UID: \"c85436c5-ec45-4464-bf21-418ee202897a\") " pod="kuadrant-system/authorino-64b5794f6c-9cvtw"
Sep 06 19:37:04.509747 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:04.509754    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n272d\" (UniqueName: \"kubernetes.io/projected/c85436c5-ec45-4464-bf21-418ee202897a-kube-api-access-n272d\") pod \"authorino-64b5794f6c-9cvtw\" (UID: \"c85436c5-ec45-4464-bf21-418ee202897a\") " pod="kuadrant-system/authorino-64b5794f6c-9cvtw"
Sep 06 19:37:04.512223 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:04.512198    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c85436c5-ec45-4464-bf21-418ee202897a-tls-cert\") pod \"authorino-64b5794f6c-9cvtw\" (UID: \"c85436c5-ec45-4464-bf21-418ee202897a\") " pod="kuadrant-system/authorino-64b5794f6c-9cvtw"
Sep 06 19:37:04.516777 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:04.516758    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n272d\" (UniqueName: \"kubernetes.io/projected/c85436c5-ec45-4464-bf21-418ee202897a-kube-api-access-n272d\") pod \"authorino-64b5794f6c-9cvtw\" (UID: \"c85436c5-ec45-4464-bf21-418ee202897a\") " pod="kuadrant-system/authorino-64b5794f6c-9cvtw"
Sep 06 19:37:04.617626 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:04.617589    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b5794f6c-9cvtw"
Sep 06 19:37:04.884223 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:04.884185    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64b5794f6c-9cvtw"]
Sep 06 19:37:04.886666 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:37:04.886642    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc85436c5_ec45_4464_bf21_418ee202897a.slice/crio-bea669083c358125bb824c01e208b25df7b0b1609e9de6633acde6230e972e2d WatchSource:0}: Error finding container bea669083c358125bb824c01e208b25df7b0b1609e9de6633acde6230e972e2d: Status 404 returned error can't find the container with id bea669083c358125bb824c01e208b25df7b0b1609e9de6633acde6230e972e2d
Sep 06 19:37:05.597563 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:05.597526    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64b5794f6c-9cvtw" event={"ID":"c85436c5-ec45-4464-bf21-418ee202897a","Type":"ContainerStarted","Data":"ffc53db0161a5bdcedcec39cbcfbb992bfb0ea0ed0dcceacac7d8e77f097dac4"}
Sep 06 19:37:05.597563 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:05.597568    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64b5794f6c-9cvtw" event={"ID":"c85436c5-ec45-4464-bf21-418ee202897a","Type":"ContainerStarted","Data":"bea669083c358125bb824c01e208b25df7b0b1609e9de6633acde6230e972e2d"}
Sep 06 19:37:05.615126 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:05.615077    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-64b5794f6c-9cvtw" podStartSLOduration=1.196435695 podStartE2EDuration="1.615061708s" totalImagesPullingTime="418.626013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:37:05.613861485 +0000 UTC m=+589.184159831" watchObservedRunningTime="2026-09-06 19:37:05.615061708 +0000 UTC m=+589.185360056"
Sep 06 19:37:05.638212 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:05.638174    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-hnsbn"]
Sep 06 19:37:05.638440 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:05.638404    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-hnsbn" podUID="0b9625a5-9fbb-4d84-91de-958118635e9a" containerName="authorino" containerID="cri-o://8986a68625ccb47f28e29e96383b1ca2bd8b5dad3922315c8aca762947cd3984" gracePeriod=30
Sep 06 19:37:05.887933 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:05.887909    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-hnsbn"
Sep 06 19:37:06.024647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:06.024611    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mk2r\" (UniqueName: \"kubernetes.io/projected/0b9625a5-9fbb-4d84-91de-958118635e9a-kube-api-access-9mk2r\") pod \"0b9625a5-9fbb-4d84-91de-958118635e9a\" (UID: \"0b9625a5-9fbb-4d84-91de-958118635e9a\") "
Sep 06 19:37:06.026665 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:06.026636    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9625a5-9fbb-4d84-91de-958118635e9a-kube-api-access-9mk2r" (OuterVolumeSpecName: "kube-api-access-9mk2r") pod "0b9625a5-9fbb-4d84-91de-958118635e9a" (UID: "0b9625a5-9fbb-4d84-91de-958118635e9a"). InnerVolumeSpecName "kube-api-access-9mk2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:37:06.126321 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:06.126243    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mk2r\" (UniqueName: \"kubernetes.io/projected/0b9625a5-9fbb-4d84-91de-958118635e9a-kube-api-access-9mk2r\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:37:06.603776 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:06.603742    2571 generic.go:356] "Generic (PLEG): container finished" podID="0b9625a5-9fbb-4d84-91de-958118635e9a" containerID="8986a68625ccb47f28e29e96383b1ca2bd8b5dad3922315c8aca762947cd3984" exitCode=0
Sep 06 19:37:06.604263 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:06.603790    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-hnsbn"
Sep 06 19:37:06.604263 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:06.603823    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-hnsbn" event={"ID":"0b9625a5-9fbb-4d84-91de-958118635e9a","Type":"ContainerDied","Data":"8986a68625ccb47f28e29e96383b1ca2bd8b5dad3922315c8aca762947cd3984"}
Sep 06 19:37:06.604263 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:06.603856    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-hnsbn" event={"ID":"0b9625a5-9fbb-4d84-91de-958118635e9a","Type":"ContainerDied","Data":"f38c839cbbf517faa30957e2f32c53c3e21b4f9fcec90203c0e8e41e8c2507c6"}
Sep 06 19:37:06.604263 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:06.603881    2571 scope.go:117] "RemoveContainer" containerID="8986a68625ccb47f28e29e96383b1ca2bd8b5dad3922315c8aca762947cd3984"
Sep 06 19:37:06.612185 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:06.612166    2571 scope.go:117] "RemoveContainer" containerID="8986a68625ccb47f28e29e96383b1ca2bd8b5dad3922315c8aca762947cd3984"
Sep 06 19:37:06.612460 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:37:06.612443    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8986a68625ccb47f28e29e96383b1ca2bd8b5dad3922315c8aca762947cd3984\": container with ID starting with 8986a68625ccb47f28e29e96383b1ca2bd8b5dad3922315c8aca762947cd3984 not found: ID does not exist" containerID="8986a68625ccb47f28e29e96383b1ca2bd8b5dad3922315c8aca762947cd3984"
Sep 06 19:37:06.612507 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:06.612468    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8986a68625ccb47f28e29e96383b1ca2bd8b5dad3922315c8aca762947cd3984"} err="failed to get container status \"8986a68625ccb47f28e29e96383b1ca2bd8b5dad3922315c8aca762947cd3984\": rpc error: code = NotFound desc = could not find container \"8986a68625ccb47f28e29e96383b1ca2bd8b5dad3922315c8aca762947cd3984\": container with ID starting with 8986a68625ccb47f28e29e96383b1ca2bd8b5dad3922315c8aca762947cd3984 not found: ID does not exist"
Sep 06 19:37:06.625074 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:06.625052    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-hnsbn"]
Sep 06 19:37:06.628552 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:06.628523    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-hnsbn"]
Sep 06 19:37:06.918950 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:06.918870    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b9625a5-9fbb-4d84-91de-958118635e9a" path="/var/lib/kubelet/pods/0b9625a5-9fbb-4d84-91de-958118635e9a/volumes"
Sep 06 19:37:16.860013 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:16.859983    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 19:37:16.860852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:16.860829    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 19:37:48.872640 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:48.872605    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:37:56.435154 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:56.435106    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-95l97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 06 19:37:56.435588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:56.435210    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97" podUID="c7c1898a-3846-4250-8601-fbc9bb7fcc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 06 19:37:57.837626 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:57.837597    2571 generic.go:356] "Generic (PLEG): container finished" podID="c7c1898a-3846-4250-8601-fbc9bb7fcc70" containerID="9f47d0bc9e3eeb8c871a09f38f008125f202867a43aa514c16b40f4dcb95843c" exitCode=0
Sep 06 19:37:57.838052 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:57.837668    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97" event={"ID":"c7c1898a-3846-4250-8601-fbc9bb7fcc70","Type":"ContainerDied","Data":"9f47d0bc9e3eeb8c871a09f38f008125f202867a43aa514c16b40f4dcb95843c"}
Sep 06 19:37:57.838099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:57.838046    2571 scope.go:117] "RemoveContainer" containerID="9f47d0bc9e3eeb8c871a09f38f008125f202867a43aa514c16b40f4dcb95843c"
Sep 06 19:37:57.839066 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:57.839040    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:37:57.839117 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:57.839105    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:37:57.839156 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:57.839138    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:37:58.852017 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:58.851976    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97" event={"ID":"c7c1898a-3846-4250-8601-fbc9bb7fcc70","Type":"ContainerStarted","Data":"8289f29041703d81f097e27aa2b5ef9aebe6f19a0167dbc3a83b4a8e3a1f38e5"}
Sep 06 19:37:58.852622 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:58.852598    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:37:58.852860 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:58.852833    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-95l97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Sep 06 19:37:58.852942 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:58.852893    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97" podUID="c7c1898a-3846-4250-8601-fbc9bb7fcc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Sep 06 19:37:59.853288 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:59.853249    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-95l97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Sep 06 19:37:59.853893 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:37:59.853845    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97" podUID="c7c1898a-3846-4250-8601-fbc9bb7fcc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Sep 06 19:38:00.853279 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:38:00.853234    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-95l97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Sep 06 19:38:00.853570 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:38:00.853536    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97" podUID="c7c1898a-3846-4250-8601-fbc9bb7fcc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Sep 06 19:38:01.853337 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:38:01.853289    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-95l97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Sep 06 19:38:01.853556 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:38:01.853361    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97" podUID="c7c1898a-3846-4250-8601-fbc9bb7fcc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Sep 06 19:38:02.853365 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:38:02.853315    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-95l97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Sep 06 19:38:02.853858 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:38:02.853406    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97" podUID="c7c1898a-3846-4250-8601-fbc9bb7fcc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Sep 06 19:38:03.857077 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:38:03.857049    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:38:03.857428 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:38:03.857305    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:38:03.857945 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:38:03.857928    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-95l97"
Sep 06 19:38:07.646629 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:38:07.646588    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:38:14.434014 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:38:14.433976    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:38:34.543568 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:38:34.543529    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:38:53.436285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:38:53.436245    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:39:08.326196 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.326158    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"]
Sep 06 19:39:08.326698 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.326679    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b9625a5-9fbb-4d84-91de-958118635e9a" containerName="authorino"
Sep 06 19:39:08.326742 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.326703    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b9625a5-9fbb-4d84-91de-958118635e9a" containerName="authorino"
Sep 06 19:39:08.326819 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.326807    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b9625a5-9fbb-4d84-91de-958118635e9a" containerName="authorino"
Sep 06 19:39:08.330007 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.329970    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.333231 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.333207    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ldj7j\""
Sep 06 19:39:08.333384 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.333218    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 06 19:39:08.333457 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.333228    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 06 19:39:08.333513 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.333258    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 06 19:39:08.340128 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.340101    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"]
Sep 06 19:39:08.433015 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.432974    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99fa825-2680-4d0e-b9b4-abe3f01c967d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.433015 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.433019    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99fa825-2680-4d0e-b9b4-abe3f01c967d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.433235 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.433055    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99fa825-2680-4d0e-b9b4-abe3f01c967d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.433235 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.433075    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv88b\" (UniqueName: \"kubernetes.io/projected/b99fa825-2680-4d0e-b9b4-abe3f01c967d-kube-api-access-xv88b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.433235 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.433165    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99fa825-2680-4d0e-b9b4-abe3f01c967d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.433235 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.433197    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99fa825-2680-4d0e-b9b4-abe3f01c967d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.534375 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.534338    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99fa825-2680-4d0e-b9b4-abe3f01c967d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.534375 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.534379    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99fa825-2680-4d0e-b9b4-abe3f01c967d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.534635 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.534412    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99fa825-2680-4d0e-b9b4-abe3f01c967d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.534635 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.534431    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv88b\" (UniqueName: \"kubernetes.io/projected/b99fa825-2680-4d0e-b9b4-abe3f01c967d-kube-api-access-xv88b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.534635 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.534467    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99fa825-2680-4d0e-b9b4-abe3f01c967d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.534635 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.534482    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99fa825-2680-4d0e-b9b4-abe3f01c967d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.534869 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.534842    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99fa825-2680-4d0e-b9b4-abe3f01c967d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.534918 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.534885    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99fa825-2680-4d0e-b9b4-abe3f01c967d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.534918 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.534902    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99fa825-2680-4d0e-b9b4-abe3f01c967d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.536748 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.536720    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99fa825-2680-4d0e-b9b4-abe3f01c967d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.537055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.537034    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99fa825-2680-4d0e-b9b4-abe3f01c967d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.541858 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.541821    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv88b\" (UniqueName: \"kubernetes.io/projected/b99fa825-2680-4d0e-b9b4-abe3f01c967d-kube-api-access-xv88b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8\" (UID: \"b99fa825-2680-4d0e-b9b4-abe3f01c967d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.641912 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.641806    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:08.916153 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:39:08.916120    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99fa825_2680_4d0e_b9b4_abe3f01c967d.slice/crio-cc789114afea71535fe29a561a82ed27803d4632a1cfa01746f63c5dfe4edb2d WatchSource:0}: Error finding container cc789114afea71535fe29a561a82ed27803d4632a1cfa01746f63c5dfe4edb2d: Status 404 returned error can't find the container with id cc789114afea71535fe29a561a82ed27803d4632a1cfa01746f63c5dfe4edb2d
Sep 06 19:39:08.917789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.917769    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 19:39:08.919803 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:08.919780    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"]
Sep 06 19:39:09.180115 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:09.180031    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8" event={"ID":"b99fa825-2680-4d0e-b9b4-abe3f01c967d","Type":"ContainerStarted","Data":"cc789114afea71535fe29a561a82ed27803d4632a1cfa01746f63c5dfe4edb2d"}
Sep 06 19:39:09.742219 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:09.742176    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:39:10.186662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:10.186624    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8" event={"ID":"b99fa825-2680-4d0e-b9b4-abe3f01c967d","Type":"ContainerStarted","Data":"8857e425461c21c19d256cd445da4422f6ed29db8d0662fa0d36ff9d67403220"}
Sep 06 19:39:10.206032 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:10.205984    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8" podStartSLOduration=1.265466726 podStartE2EDuration="2.205969302s" totalImagesPullingTime="940.502576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:39:10.204247639 +0000 UTC m=+713.774545986" watchObservedRunningTime="2026-09-06 19:39:10.205969302 +0000 UTC m=+713.776267676"
Sep 06 19:39:18.642140 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:18.642098    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:18.642621 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:18.642165    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:18.654349 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:18.654323    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:19.241743 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:19.241711    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8"
Sep 06 19:39:19.544914 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:19.544824    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:39:44.638653 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.638618    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"]
Sep 06 19:39:44.642299 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.642279    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.644662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.644637    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 06 19:39:44.652230 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.652206    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"]
Sep 06 19:39:44.656639 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.656614    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.656750 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.656657    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.656750 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.656682    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.656750 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.656698    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbrnq\" (UniqueName: \"kubernetes.io/projected/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-kube-api-access-xbrnq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.656869 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.656756    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.656916 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.656892    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.758108 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.758076    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.758108 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.758113    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.758350 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.758145    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.758350 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.758178    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.758350 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.758204    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbrnq\" (UniqueName: \"kubernetes.io/projected/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-kube-api-access-xbrnq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.758350 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.758230    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.758601 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.758549    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.758711 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.758611    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.758711 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.758630    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.760407 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.760385    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.760692 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.760670    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.765440 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.765421    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbrnq\" (UniqueName: \"kubernetes.io/projected/e7fd76f8-8e82-4d03-9bbc-ef42b3b29482-kube-api-access-xbrnq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-llp2n\" (UID: \"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:44.951734 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:44.951622    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:45.223768 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:45.223742    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"]
Sep 06 19:39:45.226200 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:39:45.226173    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7fd76f8_8e82_4d03_9bbc_ef42b3b29482.slice/crio-e4d9277305f5403cd153e0d050ef18b55189326a9beb8cbba0f8e336c18a3983 WatchSource:0}: Error finding container e4d9277305f5403cd153e0d050ef18b55189326a9beb8cbba0f8e336c18a3983: Status 404 returned error can't find the container with id e4d9277305f5403cd153e0d050ef18b55189326a9beb8cbba0f8e336c18a3983
Sep 06 19:39:45.358336 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:45.358292    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n" event={"ID":"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482","Type":"ContainerStarted","Data":"e4d9277305f5403cd153e0d050ef18b55189326a9beb8cbba0f8e336c18a3983"}
Sep 06 19:39:46.364473 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:46.364435    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n" event={"ID":"e7fd76f8-8e82-4d03-9bbc-ef42b3b29482","Type":"ContainerStarted","Data":"9a5f8855d43fb7da04ac8ee073816b23329414ae88feb6468f4633c73abf0d99"}
Sep 06 19:39:46.380119 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:46.380070    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n" podStartSLOduration=2.201585975 podStartE2EDuration="2.38005501s" totalImagesPullingTime="178.469035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:39:46.379655929 +0000 UTC m=+749.949954274" watchObservedRunningTime="2026-09-06 19:39:46.38005501 +0000 UTC m=+749.950353355"
Sep 06 19:39:46.840305 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:46.840266    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:39:54.952737 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:54.952687    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:54.952737 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:54.952743    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:54.972616 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:54.972580    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:39:55.420920 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:39:55.420891    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-llp2n"
Sep 06 19:41:01.436016 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:01.435974    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6584bfcdf4-97x8k"]
Sep 06 19:41:01.439412 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:01.439393    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6584bfcdf4-97x8k"
Sep 06 19:41:01.446879 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:01.446851    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6584bfcdf4-97x8k"]
Sep 06 19:41:01.547747 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:01.547702    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5-tls-cert\") pod \"authorino-6584bfcdf4-97x8k\" (UID: \"d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5\") " pod="kuadrant-system/authorino-6584bfcdf4-97x8k"
Sep 06 19:41:01.547747 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:01.547752    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrz4m\" (UniqueName: \"kubernetes.io/projected/d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5-kube-api-access-zrz4m\") pod \"authorino-6584bfcdf4-97x8k\" (UID: \"d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5\") " pod="kuadrant-system/authorino-6584bfcdf4-97x8k"
Sep 06 19:41:01.649195 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:01.649155    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5-tls-cert\") pod \"authorino-6584bfcdf4-97x8k\" (UID: \"d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5\") " pod="kuadrant-system/authorino-6584bfcdf4-97x8k"
Sep 06 19:41:01.649195 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:01.649197    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrz4m\" (UniqueName: \"kubernetes.io/projected/d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5-kube-api-access-zrz4m\") pod \"authorino-6584bfcdf4-97x8k\" (UID: \"d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5\") " pod="kuadrant-system/authorino-6584bfcdf4-97x8k"
Sep 06 19:41:01.651646 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:01.651610    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5-tls-cert\") pod \"authorino-6584bfcdf4-97x8k\" (UID: \"d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5\") " pod="kuadrant-system/authorino-6584bfcdf4-97x8k"
Sep 06 19:41:01.656768 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:01.656746    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrz4m\" (UniqueName: \"kubernetes.io/projected/d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5-kube-api-access-zrz4m\") pod \"authorino-6584bfcdf4-97x8k\" (UID: \"d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5\") " pod="kuadrant-system/authorino-6584bfcdf4-97x8k"
Sep 06 19:41:01.749141 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:01.749050    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6584bfcdf4-97x8k"
Sep 06 19:41:02.016522 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:02.016490    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6584bfcdf4-97x8k"]
Sep 06 19:41:02.019045 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:41:02.019009    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1f4450a_c494_4f8a_b0ce_7d1ded4d6bd5.slice/crio-896c892610cd4a55ba2079699c0c43e70dfbcdd596026dba1b6da9398350918e WatchSource:0}: Error finding container 896c892610cd4a55ba2079699c0c43e70dfbcdd596026dba1b6da9398350918e: Status 404 returned error can't find the container with id 896c892610cd4a55ba2079699c0c43e70dfbcdd596026dba1b6da9398350918e
Sep 06 19:41:02.723876 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:02.723838    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6584bfcdf4-97x8k" event={"ID":"d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5","Type":"ContainerStarted","Data":"896c892610cd4a55ba2079699c0c43e70dfbcdd596026dba1b6da9398350918e"}
Sep 06 19:41:03.738813 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:03.738767    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6584bfcdf4-97x8k" event={"ID":"d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5","Type":"ContainerStarted","Data":"072d9bd0b9519942600ff5bcdb65ffc96ca2c7751a0d3b532a045705306aa040"}
Sep 06 19:41:03.756318 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:03.754996    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6584bfcdf4-97x8k" podStartSLOduration=1.985751434 podStartE2EDuration="2.754975485s" totalImagesPullingTime="769.224051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:41:03.753356635 +0000 UTC m=+827.323654982" watchObservedRunningTime="2026-09-06 19:41:03.754975485 +0000 UTC m=+827.325273831"
Sep 06 19:41:03.784088 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:03.784045    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64b5794f6c-9cvtw"]
Sep 06 19:41:03.784911 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:03.784679    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-64b5794f6c-9cvtw" podUID="c85436c5-ec45-4464-bf21-418ee202897a" containerName="authorino" containerID="cri-o://ffc53db0161a5bdcedcec39cbcfbb992bfb0ea0ed0dcceacac7d8e77f097dac4" gracePeriod=30
Sep 06 19:41:04.049297 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.049267    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b5794f6c-9cvtw"
Sep 06 19:41:04.181841 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.181800    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n272d\" (UniqueName: \"kubernetes.io/projected/c85436c5-ec45-4464-bf21-418ee202897a-kube-api-access-n272d\") pod \"c85436c5-ec45-4464-bf21-418ee202897a\" (UID: \"c85436c5-ec45-4464-bf21-418ee202897a\") "
Sep 06 19:41:04.182071 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.181912    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c85436c5-ec45-4464-bf21-418ee202897a-tls-cert\") pod \"c85436c5-ec45-4464-bf21-418ee202897a\" (UID: \"c85436c5-ec45-4464-bf21-418ee202897a\") "
Sep 06 19:41:04.184130 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.184095    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85436c5-ec45-4464-bf21-418ee202897a-kube-api-access-n272d" (OuterVolumeSpecName: "kube-api-access-n272d") pod "c85436c5-ec45-4464-bf21-418ee202897a" (UID: "c85436c5-ec45-4464-bf21-418ee202897a"). InnerVolumeSpecName "kube-api-access-n272d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:41:04.192575 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.192540    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85436c5-ec45-4464-bf21-418ee202897a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c85436c5-ec45-4464-bf21-418ee202897a" (UID: "c85436c5-ec45-4464-bf21-418ee202897a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:41:04.283018 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.282923    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n272d\" (UniqueName: \"kubernetes.io/projected/c85436c5-ec45-4464-bf21-418ee202897a-kube-api-access-n272d\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:41:04.283018 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.282968    2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c85436c5-ec45-4464-bf21-418ee202897a-tls-cert\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:41:04.745473 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.745433    2571 generic.go:356] "Generic (PLEG): container finished" podID="c85436c5-ec45-4464-bf21-418ee202897a" containerID="ffc53db0161a5bdcedcec39cbcfbb992bfb0ea0ed0dcceacac7d8e77f097dac4" exitCode=0
Sep 06 19:41:04.745889 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.745480    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64b5794f6c-9cvtw"
Sep 06 19:41:04.745889 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.745518    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64b5794f6c-9cvtw" event={"ID":"c85436c5-ec45-4464-bf21-418ee202897a","Type":"ContainerDied","Data":"ffc53db0161a5bdcedcec39cbcfbb992bfb0ea0ed0dcceacac7d8e77f097dac4"}
Sep 06 19:41:04.745889 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.745561    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64b5794f6c-9cvtw" event={"ID":"c85436c5-ec45-4464-bf21-418ee202897a","Type":"ContainerDied","Data":"bea669083c358125bb824c01e208b25df7b0b1609e9de6633acde6230e972e2d"}
Sep 06 19:41:04.745889 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.745579    2571 scope.go:117] "RemoveContainer" containerID="ffc53db0161a5bdcedcec39cbcfbb992bfb0ea0ed0dcceacac7d8e77f097dac4"
Sep 06 19:41:04.754543 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.754519    2571 scope.go:117] "RemoveContainer" containerID="ffc53db0161a5bdcedcec39cbcfbb992bfb0ea0ed0dcceacac7d8e77f097dac4"
Sep 06 19:41:04.754824 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:41:04.754806    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffc53db0161a5bdcedcec39cbcfbb992bfb0ea0ed0dcceacac7d8e77f097dac4\": container with ID starting with ffc53db0161a5bdcedcec39cbcfbb992bfb0ea0ed0dcceacac7d8e77f097dac4 not found: ID does not exist" containerID="ffc53db0161a5bdcedcec39cbcfbb992bfb0ea0ed0dcceacac7d8e77f097dac4"
Sep 06 19:41:04.754875 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.754835    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffc53db0161a5bdcedcec39cbcfbb992bfb0ea0ed0dcceacac7d8e77f097dac4"} err="failed to get container status \"ffc53db0161a5bdcedcec39cbcfbb992bfb0ea0ed0dcceacac7d8e77f097dac4\": rpc error: code = NotFound desc = could not find container \"ffc53db0161a5bdcedcec39cbcfbb992bfb0ea0ed0dcceacac7d8e77f097dac4\": container with ID starting with ffc53db0161a5bdcedcec39cbcfbb992bfb0ea0ed0dcceacac7d8e77f097dac4 not found: ID does not exist"
Sep 06 19:41:04.766386 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.766356    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64b5794f6c-9cvtw"]
Sep 06 19:41:04.772558 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.772523    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-64b5794f6c-9cvtw"]
Sep 06 19:41:04.918303 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:04.918272    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c85436c5-ec45-4464-bf21-418ee202897a" path="/var/lib/kubelet/pods/c85436c5-ec45-4464-bf21-418ee202897a/volumes"
Sep 06 19:41:23.954028 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:23.953975    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:41:28.402126 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.402084    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"]
Sep 06 19:41:28.402727 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.402481    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c85436c5-ec45-4464-bf21-418ee202897a" containerName="authorino"
Sep 06 19:41:28.402727 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.402493    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c85436c5-ec45-4464-bf21-418ee202897a" containerName="authorino"
Sep 06 19:41:28.402727 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.402590    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c85436c5-ec45-4464-bf21-418ee202897a" containerName="authorino"
Sep 06 19:41:28.407080 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.407055    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.409630 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.409605    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4e3f8e3b-openshift-default-dockercfg-mzfwf\""
Sep 06 19:41:28.416108 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.416029    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"]
Sep 06 19:41:28.521014 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.520946    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istiod-ca-cert\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.521204 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.521021    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-data\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.521204 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.521058    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-envoy\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.521204 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.521083    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mwdc\" (UniqueName: \"kubernetes.io/projected/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-kube-api-access-7mwdc\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.521204 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.521155    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-token\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.521419 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.521212    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-podinfo\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.521419 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.521287    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-credential-socket\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.521419 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.521326    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-workload-socket\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.521419 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.521371    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-workload-certs\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.622600 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.622557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-envoy\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.622600 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.622608    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mwdc\" (UniqueName: \"kubernetes.io/projected/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-kube-api-access-7mwdc\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.622878 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.622640    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-token\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.622878 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.622667    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-podinfo\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.622878 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.622727    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-credential-socket\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.622878 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.622754    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-workload-socket\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.623128 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.622888    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-workload-certs\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.623128 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.622935    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istiod-ca-cert\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.623128 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.622978    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-data\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.623274 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.623171    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-credential-socket\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.623355 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.623333    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-workload-socket\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.623815 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.623783    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istiod-ca-cert\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.623815 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.623805    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-workload-certs\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.624074 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.623801    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-data\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.625100 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.625076    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-envoy\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.625390 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.625370    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-podinfo\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.629998 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.629947    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-token\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.630200 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.630177    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mwdc\" (UniqueName: \"kubernetes.io/projected/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-kube-api-access-7mwdc\") pod \"e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") " pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.721219 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.721130    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:28.802111 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.802073    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"]
Sep 06 19:41:28.806793 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.806352    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:28.808823 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.808776    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-0ebbc9-openshift-default-dockercfg-nkn7s\""
Sep 06 19:41:28.816482 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.816406    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"]
Sep 06 19:41:28.928004 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.927937    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54trq\" (UniqueName: \"kubernetes.io/projected/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-kube-api-access-54trq\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:28.928194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.928012    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-workload-socket\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:28.928194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.928042    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-token\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:28.928194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.928072    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-workload-certs\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:28.928194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.928103    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-data\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:28.928194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.928174    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-podinfo\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:28.928397 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.928222    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-envoy\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:28.928397 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.928298    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istiod-ca-cert\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:28.928397 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:28.928326    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-credential-socket\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.001405 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.001369    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"]
Sep 06 19:41:29.002167 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:41:29.002139    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c2f0be_d468_4c19_ae10_ec064e9a7af7.slice/crio-20bcb0431f7b54e9c0680134c6b442693e69bdd3e0fcebdb0386a182093ee454 WatchSource:0}: Error finding container 20bcb0431f7b54e9c0680134c6b442693e69bdd3e0fcebdb0386a182093ee454: Status 404 returned error can't find the container with id 20bcb0431f7b54e9c0680134c6b442693e69bdd3e0fcebdb0386a182093ee454
Sep 06 19:41:29.004377 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.004344    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:41:29.004483 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.004418    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:41:29.004483 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.004444    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:41:29.029137 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.029107    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-credential-socket\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.029279 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.029152    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54trq\" (UniqueName: \"kubernetes.io/projected/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-kube-api-access-54trq\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.029353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.029275    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-workload-socket\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.029353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.029320    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-token\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.029460 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.029354    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-workload-certs\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.029460 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.029386    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-data\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.029567 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.029475    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-credential-socket\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.029567 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.029483    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-podinfo\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.029672 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.029563    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-workload-socket\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.029672 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.029594    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-envoy\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.029771 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.029692    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istiod-ca-cert\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.029771 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.029720    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-workload-certs\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.029771 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.029754    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-data\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.030234 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.030214    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istiod-ca-cert\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.031848 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.031825    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-envoy\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.031996 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.031976    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-podinfo\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.036391 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.036363    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-token\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.036391 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.036388    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54trq\" (UniqueName: \"kubernetes.io/projected/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-kube-api-access-54trq\") pod \"e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") " pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.126217 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.126180    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:29.403435 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.403408    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"]
Sep 06 19:41:29.405850 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:41:29.405808    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025a5d67_7c5a_4b97_9d22_c3112e7a1c94.slice/crio-3796e1aa10d991c3f78a1d3e46424452c0b8d659a477eb0e901072a670a80c34 WatchSource:0}: Error finding container 3796e1aa10d991c3f78a1d3e46424452c0b8d659a477eb0e901072a670a80c34: Status 404 returned error can't find the container with id 3796e1aa10d991c3f78a1d3e46424452c0b8d659a477eb0e901072a670a80c34
Sep 06 19:41:29.408289 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.408241    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:41:29.408402 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.408334    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:41:29.408402 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.408373    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:41:29.867627 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.867585    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s" event={"ID":"b9c2f0be-d468-4c19-ae10-ec064e9a7af7","Type":"ContainerStarted","Data":"03104b357845c7acc69fe5ae83bcae75f4ef19196d3863c491f20ff47460ba8f"}
Sep 06 19:41:29.867828 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.867651    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s" event={"ID":"b9c2f0be-d468-4c19-ae10-ec064e9a7af7","Type":"ContainerStarted","Data":"20bcb0431f7b54e9c0680134c6b442693e69bdd3e0fcebdb0386a182093ee454"}
Sep 06 19:41:29.869297 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.869272    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd" event={"ID":"025a5d67-7c5a-4b97-9d22-c3112e7a1c94","Type":"ContainerStarted","Data":"5f3765524bb43e3b1ad73a48208811c1f531f3a826057b9edcf33bab1f73ac5b"}
Sep 06 19:41:29.869416 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.869312    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd" event={"ID":"025a5d67-7c5a-4b97-9d22-c3112e7a1c94","Type":"ContainerStarted","Data":"3796e1aa10d991c3f78a1d3e46424452c0b8d659a477eb0e901072a670a80c34"}
Sep 06 19:41:29.886410 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.886358    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s" podStartSLOduration=1.886343948 podStartE2EDuration="1.886343948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:41:29.885629434 +0000 UTC m=+853.455927781" watchObservedRunningTime="2026-09-06 19:41:29.886343948 +0000 UTC m=+853.456642288"
Sep 06 19:41:29.902837 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:29.902749    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd" podStartSLOduration=1.902731144 podStartE2EDuration="1.902731144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:41:29.90199602 +0000 UTC m=+853.472294381" watchObservedRunningTime="2026-09-06 19:41:29.902731144 +0000 UTC m=+853.473029524"
Sep 06 19:41:30.126583 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:30.126490    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:30.131612 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:30.131583    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:30.721817 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:30.721780    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:30.728544 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:30.728514    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:30.875142 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:30.875110    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:30.875335 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:30.875169    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:30.876056 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:30.876033    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:30.876158 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:30.876087    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:38.250287 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:38.250244    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:41:45.529148 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.529055    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"]
Sep 06 19:41:45.533207 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.533179    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.535920 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.535883    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-6e7ee48c-kserve-self-signed-certs\""
Sep 06 19:41:45.541082 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.541051    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"]
Sep 06 19:41:45.707372 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.707326    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa9ed92-6362-422e-9ff1-6f93523ffc01-tls-certs\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.707574 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.707425    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-dshm\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.707574 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.707558    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgqsx\" (UniqueName: \"kubernetes.io/projected/afa9ed92-6362-422e-9ff1-6f93523ffc01-kube-api-access-hgqsx\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.707704 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.707619    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-home\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.707777 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.707739    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-model-cache\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.707777 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.707766    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-tmp-dir\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.809455 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.809350    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-model-cache\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.809455 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.809420    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-tmp-dir\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.809678 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.809465    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa9ed92-6362-422e-9ff1-6f93523ffc01-tls-certs\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.809678 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.809522    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-dshm\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.809678 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.809579    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgqsx\" (UniqueName: \"kubernetes.io/projected/afa9ed92-6362-422e-9ff1-6f93523ffc01-kube-api-access-hgqsx\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.809678 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.809618    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-home\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.809892 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.809861    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-model-cache\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.810269 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.810220    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-home\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.810269 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.810240    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-tmp-dir\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.812460 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.812433    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-dshm\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.812717 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.812698    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa9ed92-6362-422e-9ff1-6f93523ffc01-tls-certs\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.817768 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.817740    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgqsx\" (UniqueName: \"kubernetes.io/projected/afa9ed92-6362-422e-9ff1-6f93523ffc01-kube-api-access-hgqsx\") pod \"e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") " pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:45.845760 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:45.845723    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:46.125402 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:46.125369    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"]
Sep 06 19:41:46.129620 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:41:46.129590    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafa9ed92_6362_422e_9ff1_6f93523ffc01.slice/crio-381ab15efdc9fa5ad38da424fcfe291110e4e40c339e264c3f202ca9d16f6bfa WatchSource:0}: Error finding container 381ab15efdc9fa5ad38da424fcfe291110e4e40c339e264c3f202ca9d16f6bfa: Status 404 returned error can't find the container with id 381ab15efdc9fa5ad38da424fcfe291110e4e40c339e264c3f202ca9d16f6bfa
Sep 06 19:41:46.917161 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:46.917123    2571 patch_prober.go:28] interesting pod/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 06 19:41:46.917695 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:46.917184    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s" podUID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 06 19:41:46.917695 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:46.917242    2571 patch_prober.go:28] interesting pod/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 06 19:41:46.917695 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:46.917289    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd" podUID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 06 19:41:46.968134 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:46.968091    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc" event={"ID":"afa9ed92-6362-422e-9ff1-6f93523ffc01","Type":"ContainerStarted","Data":"84ade560a206976191fdb7cdc0accaa829ca39440df8831dff984cb40fd929ca"}
Sep 06 19:41:46.968134 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:46.968139    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc" event={"ID":"afa9ed92-6362-422e-9ff1-6f93523ffc01","Type":"ContainerStarted","Data":"381ab15efdc9fa5ad38da424fcfe291110e4e40c339e264c3f202ca9d16f6bfa"}
Sep 06 19:41:46.985911 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:46.985854    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc" podStartSLOduration=1.9858354280000001 podStartE2EDuration="1.985835428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:41:46.984877969 +0000 UTC m=+870.555176316" watchObservedRunningTime="2026-09-06 19:41:46.985835428 +0000 UTC m=+870.556133774"
Sep 06 19:41:48.980084 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:48.980051    2571 generic.go:356] "Generic (PLEG): container finished" podID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" containerID="03104b357845c7acc69fe5ae83bcae75f4ef19196d3863c491f20ff47460ba8f" exitCode=0
Sep 06 19:41:48.980527 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:48.980118    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s" event={"ID":"b9c2f0be-d468-4c19-ae10-ec064e9a7af7","Type":"ContainerDied","Data":"03104b357845c7acc69fe5ae83bcae75f4ef19196d3863c491f20ff47460ba8f"}
Sep 06 19:41:48.980607 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:48.980578    2571 scope.go:117] "RemoveContainer" containerID="03104b357845c7acc69fe5ae83bcae75f4ef19196d3863c491f20ff47460ba8f"
Sep 06 19:41:48.981769 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:48.981741    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:41:48.981835 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:48.981770    2571 generic.go:356] "Generic (PLEG): container finished" podID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerID="5f3765524bb43e3b1ad73a48208811c1f531f3a826057b9edcf33bab1f73ac5b" exitCode=0
Sep 06 19:41:48.981835 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:48.981804    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd" event={"ID":"025a5d67-7c5a-4b97-9d22-c3112e7a1c94","Type":"ContainerDied","Data":"5f3765524bb43e3b1ad73a48208811c1f531f3a826057b9edcf33bab1f73ac5b"}
Sep 06 19:41:48.981835 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:48.981810    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:41:48.981934 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:48.981842    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:41:48.982119 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:48.982107    2571 scope.go:117] "RemoveContainer" containerID="5f3765524bb43e3b1ad73a48208811c1f531f3a826057b9edcf33bab1f73ac5b"
Sep 06 19:41:48.983140 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:48.983113    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:41:48.983209 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:48.983177    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:41:48.983259 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:48.983207    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:41:49.181980 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.181598    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"]
Sep 06 19:41:49.185211 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.184717    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.189247 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.189228    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d5bb2d-openshift-default-dockercfg-4rn8g\""
Sep 06 19:41:49.197927 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.197898    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"]
Sep 06 19:41:49.250304 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.250202    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-credential-socket\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.250468 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.250307    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-workload-certs\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.250468 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.250358    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-envoy\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.250468 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.250396    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-podinfo\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.250468 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.250422    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istiod-ca-cert\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.250468 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.250451    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh764\" (UniqueName: \"kubernetes.io/projected/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-kube-api-access-mh764\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.250728 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.250533    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-workload-socket\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.250728 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.250576    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-data\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.250728 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.250608    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-token\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.351867 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.351827    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-credential-socket\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.352055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.351883    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-workload-certs\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.352055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.351917    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-envoy\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.352055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.351950    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-podinfo\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.352328 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.352302    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-credential-socket\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.352525 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.352503    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-workload-certs\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.352792 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.352768    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istiod-ca-cert\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.352929 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.352915    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh764\" (UniqueName: \"kubernetes.io/projected/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-kube-api-access-mh764\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.353128 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.353113    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-workload-socket\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.353272 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.352924    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istiod-ca-cert\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.353344 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.353249    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-data\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.353344 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.353327    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-token\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.353789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.353759    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-data\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.353916 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.353851    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-workload-socket\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.354620 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.354580    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-envoy\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.354724 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.354692    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-podinfo\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.360954 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.360927    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-token\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.361798 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.361774    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh764\" (UniqueName: \"kubernetes.io/projected/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-kube-api-access-mh764\") pod \"e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") " pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.503061 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.502942    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:49.832761 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.832729    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"]
Sep 06 19:41:49.839803 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.839745    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:41:49.840118 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.840079    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:41:49.840363 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.840327    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:41:49.996154 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.996116    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s" event={"ID":"b9c2f0be-d468-4c19-ae10-ec064e9a7af7","Type":"ContainerStarted","Data":"91d7655eb2cda847524875fce79cac131734f37b939f3ca6d4d0192b32ffc5fe"}
Sep 06 19:41:49.998338 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.998242    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq" event={"ID":"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d","Type":"ContainerStarted","Data":"1ddd37391e7f4ec28a6ad89893f7e1a7fdeffbb7d75ba2dcb9ca887837a7e484"}
Sep 06 19:41:49.998338 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:49.998298    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq" event={"ID":"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d","Type":"ContainerStarted","Data":"1ad233ed379a5735df7edee301c2bbac359a1d8acea378abea16b8a18a0c8ad6"}
Sep 06 19:41:50.001189 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:50.001067    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd" event={"ID":"025a5d67-7c5a-4b97-9d22-c3112e7a1c94","Type":"ContainerStarted","Data":"105dfc3ffadfb893f49e6d9bb61f7dd046444e038b17b7a3dd91b0939d8a22d8"}
Sep 06 19:41:50.034934 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:50.034870    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq" podStartSLOduration=1.034846599 podStartE2EDuration="1.034846599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:41:50.033689332 +0000 UTC m=+873.603987679" watchObservedRunningTime="2026-09-06 19:41:50.034846599 +0000 UTC m=+873.605144946"
Sep 06 19:41:50.127300 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:50.127200    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:50.127499 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:50.127472    2571 patch_prober.go:28] interesting pod/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 06 19:41:50.127581 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:50.127533    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd" podUID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 06 19:41:50.503265 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:50.503169    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:50.504880 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:50.504850    2571 patch_prober.go:28] interesting pod/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 06 19:41:50.505024 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:50.504913    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq" podUID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 06 19:41:50.721802 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:50.721763    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:50.722137 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:50.722106    2571 patch_prober.go:28] interesting pod/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 06 19:41:50.722255 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:50.722163    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s" podUID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 06 19:41:51.127595 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:51.127557    2571 patch_prober.go:28] interesting pod/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 06 19:41:51.128126 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:51.127625    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd" podUID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 06 19:41:51.507982 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:51.507869    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:51.722182 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:51.722144    2571 patch_prober.go:28] interesting pod/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 06 19:41:51.722364 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:51.722201    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s" podUID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 06 19:41:52.010771 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:52.010732    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:52.011728 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:52.011706    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:41:52.127089 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:52.127059    2571 patch_prober.go:28] interesting pod/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 06 19:41:52.127301 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:52.127279    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd" podUID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 06 19:41:52.721850 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:52.721807    2571 patch_prober.go:28] interesting pod/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 06 19:41:52.722329 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:52.721881    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s" podUID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 06 19:41:53.126758 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:53.126721    2571 patch_prober.go:28] interesting pod/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 06 19:41:53.126978 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:53.126781    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd" podUID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 06 19:41:53.721763 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:53.721730    2571 patch_prober.go:28] interesting pod/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 06 19:41:53.721929 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:53.721788    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s" podUID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 06 19:41:54.127022 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:54.126984    2571 patch_prober.go:28] interesting pod/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 06 19:41:54.127219 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:54.127043    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd" podUID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 06 19:41:54.451368 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:54.451263    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:41:54.722617 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:54.722531    2571 patch_prober.go:28] interesting pod/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 06 19:41:54.722617 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:54.722586    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s" podUID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 06 19:41:55.126813 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:55.126781    2571 patch_prober.go:28] interesting pod/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 06 19:41:55.127016 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:55.126834    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd" podUID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 06 19:41:55.726403 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:55.726367    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:55.726849 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:55.726780    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:55.727448 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:55.727429    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:41:55.846988 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:55.846933    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:55.847332 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:55.847304    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:55.865265 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:55.865229    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:56.045273 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:56.045187    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:41:56.131404 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:56.131371    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:56.131761 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:56.131739    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:56.132530 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:56.132507    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:41:56.642002 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:56.641972    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr"]
Sep 06 19:41:56.644636 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:56.644618    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr"
Sep 06 19:41:56.647081 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:56.647064    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h45ww\""
Sep 06 19:41:56.647159 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:56.647131    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 19:41:56.651535 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:56.651511    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr"]
Sep 06 19:41:56.743452 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:56.743413    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a6445-4965-4fdf-9c19-e16cfa3caad2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr\" (UID: \"3d6a6445-4965-4fdf-9c19-e16cfa3caad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr"
Sep 06 19:41:56.844074 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:56.844039    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a6445-4965-4fdf-9c19-e16cfa3caad2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr\" (UID: \"3d6a6445-4965-4fdf-9c19-e16cfa3caad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr"
Sep 06 19:41:56.844720 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:56.844699    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a6445-4965-4fdf-9c19-e16cfa3caad2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr\" (UID: \"3d6a6445-4965-4fdf-9c19-e16cfa3caad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr"
Sep 06 19:41:56.957354 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:56.957273    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr"
Sep 06 19:41:57.225795 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:57.225768    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr"]
Sep 06 19:41:57.228060 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:41:57.228028    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6a6445_4965_4fdf_9c19_e16cfa3caad2.slice/crio-6c51f470c06c3d01e7fac069525cac4d6dded784ea2773b9c22399ad1d93ec3d WatchSource:0}: Error finding container 6c51f470c06c3d01e7fac069525cac4d6dded784ea2773b9c22399ad1d93ec3d: Status 404 returned error can't find the container with id 6c51f470c06c3d01e7fac069525cac4d6dded784ea2773b9c22399ad1d93ec3d
Sep 06 19:41:58.047331 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:41:58.047256    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr" event={"ID":"3d6a6445-4965-4fdf-9c19-e16cfa3caad2","Type":"ContainerStarted","Data":"6c51f470c06c3d01e7fac069525cac4d6dded784ea2773b9c22399ad1d93ec3d"}
Sep 06 19:42:00.058815 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:00.058784    2571 generic.go:356] "Generic (PLEG): container finished" podID="3d6a6445-4965-4fdf-9c19-e16cfa3caad2" containerID="61d70076167515a14e09b2cabedd7c4a42dff163b28173ebb905cef2c0455049" exitCode=0
Sep 06 19:42:00.059214 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:00.058870    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr" event={"ID":"3d6a6445-4965-4fdf-9c19-e16cfa3caad2","Type":"ContainerDied","Data":"61d70076167515a14e09b2cabedd7c4a42dff163b28173ebb905cef2c0455049"}
Sep 06 19:42:01.199215 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:01.199187    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr"
Sep 06 19:42:01.290831 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:01.290782    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a6445-4965-4fdf-9c19-e16cfa3caad2-maas-api-service-ca\") pod \"3d6a6445-4965-4fdf-9c19-e16cfa3caad2\" (UID: \"3d6a6445-4965-4fdf-9c19-e16cfa3caad2\") "
Sep 06 19:42:01.291220 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:01.291193    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6a6445-4965-4fdf-9c19-e16cfa3caad2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d6a6445-4965-4fdf-9c19-e16cfa3caad2" (UID: "3d6a6445-4965-4fdf-9c19-e16cfa3caad2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:42:01.391844 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:01.391746    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a6445-4965-4fdf-9c19-e16cfa3caad2-maas-api-service-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:02.070189 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:02.070158    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr"
Sep 06 19:42:02.070392 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:02.070201    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr" event={"ID":"3d6a6445-4965-4fdf-9c19-e16cfa3caad2","Type":"ContainerDied","Data":"6c51f470c06c3d01e7fac069525cac4d6dded784ea2773b9c22399ad1d93ec3d"}
Sep 06 19:42:02.070392 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:02.070248    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c51f470c06c3d01e7fac069525cac4d6dded784ea2773b9c22399ad1d93ec3d"
Sep 06 19:42:03.127021 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:03.126979    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr"]
Sep 06 19:42:03.128819 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:03.128796    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4e3f8e3b-t9hlr"]
Sep 06 19:42:03.264988 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:03.264936    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"]
Sep 06 19:42:03.265250 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:03.265222    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s" podUID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" containerName="istio-proxy" containerID="cri-o://91d7655eb2cda847524875fce79cac131734f37b939f3ca6d4d0192b32ffc5fe" gracePeriod=30
Sep 06 19:42:04.209808 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.209775    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"]
Sep 06 19:42:04.210231 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.210205    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d6a6445-4965-4fdf-9c19-e16cfa3caad2" containerName="revoke-keys"
Sep 06 19:42:04.210231 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.210221    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d6a6445-4965-4fdf-9c19-e16cfa3caad2" containerName="revoke-keys"
Sep 06 19:42:04.210309 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.210300    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d6a6445-4965-4fdf-9c19-e16cfa3caad2" containerName="revoke-keys"
Sep 06 19:42:04.213630 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.213611    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.216146 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.216118    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b4020e88-gw-openshift-default-dockercfg-cslrj\""
Sep 06 19:42:04.225185 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.225123    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"]
Sep 06 19:42:04.321791 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.321746    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-data\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.321791 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.321796    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-workload-certs\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.322027 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.321914    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istiod-ca-cert\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.322027 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.321998    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-workload-socket\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.322127 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.322044    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8fxh\" (UniqueName: \"kubernetes.io/projected/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-kube-api-access-n8fxh\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.322127 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.322067    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-podinfo\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.322194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.322130    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-envoy\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.322194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.322158    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-credential-socket\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.322194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.322174    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-token\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.423368 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.423312    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istiod-ca-cert\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.423582 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.423424    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-workload-socket\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.423582 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.423478    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8fxh\" (UniqueName: \"kubernetes.io/projected/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-kube-api-access-n8fxh\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.423582 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.423518    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-podinfo\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.423582 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.423564    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-envoy\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.423810 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.423605    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-credential-socket\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.423810 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.423638    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-token\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.423810 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.423689    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-data\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.423810 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.423724    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-workload-certs\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.424095 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.423893    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-workload-socket\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.424304 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.424277    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-workload-certs\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.424460 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.424307    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istiod-ca-cert\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.424758 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.424711    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-data\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.424863 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.424763    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-credential-socket\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.426946 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.426919    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-envoy\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.427127 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.427108    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-podinfo\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.432489 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.432422    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-token\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.433376 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.433343    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8fxh\" (UniqueName: \"kubernetes.io/projected/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-kube-api-access-n8fxh\") pod \"e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") " pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.545292 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.544890    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:04.834560 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.834513    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"]
Sep 06 19:42:04.839551 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:42:04.839500    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44f184f_8a05_4478_a03b_c2e1aaa0cb4f.slice/crio-9e371a23ef10403aa01d9da50542367ad467eca6ba3046a72dbb29691d9c0b54 WatchSource:0}: Error finding container 9e371a23ef10403aa01d9da50542367ad467eca6ba3046a72dbb29691d9c0b54: Status 404 returned error can't find the container with id 9e371a23ef10403aa01d9da50542367ad467eca6ba3046a72dbb29691d9c0b54
Sep 06 19:42:04.847555 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.847472    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:42:04.847709 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.847576    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:42:04.847709 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.847627    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:42:04.919612 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:04.919583    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d6a6445-4965-4fdf-9c19-e16cfa3caad2" path="/var/lib/kubelet/pods/3d6a6445-4965-4fdf-9c19-e16cfa3caad2/volumes"
Sep 06 19:42:05.093535 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:05.093424    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8" event={"ID":"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f","Type":"ContainerStarted","Data":"55cf443d6f5b2c7ca9bc385ed2dbc540490960350838ba96cdff6a6aeaacb7e8"}
Sep 06 19:42:05.093535 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:05.093496    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8" event={"ID":"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f","Type":"ContainerStarted","Data":"9e371a23ef10403aa01d9da50542367ad467eca6ba3046a72dbb29691d9c0b54"}
Sep 06 19:42:05.114151 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:05.113804    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8" podStartSLOduration=1.113786242 podStartE2EDuration="1.113786242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:42:05.11214949 +0000 UTC m=+888.682447839" watchObservedRunningTime="2026-09-06 19:42:05.113786242 +0000 UTC m=+888.684084588"
Sep 06 19:42:05.546057 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:05.545943    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:05.559776 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:05.559737    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:06.098175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:06.098146    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:06.099075 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:06.099052    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:08.575479 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.575451    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:42:08.686496 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.686458    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-data\") pod \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") "
Sep 06 19:42:08.686703 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.686509    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istiod-ca-cert\") pod \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") "
Sep 06 19:42:08.686703 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.686553    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-podinfo\") pod \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") "
Sep 06 19:42:08.686703 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.686592    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-envoy\") pod \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") "
Sep 06 19:42:08.686703 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.686626    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-token\") pod \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") "
Sep 06 19:42:08.686703 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.686651    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-credential-socket\") pod \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") "
Sep 06 19:42:08.686703 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.686678    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-workload-socket\") pod \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") "
Sep 06 19:42:08.687048 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.686709    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mwdc\" (UniqueName: \"kubernetes.io/projected/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-kube-api-access-7mwdc\") pod \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") "
Sep 06 19:42:08.687048 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.686742    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-data" (OuterVolumeSpecName: "istio-data") pod "b9c2f0be-d468-4c19-ae10-ec064e9a7af7" (UID: "b9c2f0be-d468-4c19-ae10-ec064e9a7af7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:42:08.687048 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.686769    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-workload-certs\") pod \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\" (UID: \"b9c2f0be-d468-4c19-ae10-ec064e9a7af7\") "
Sep 06 19:42:08.687048 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.686893    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b9c2f0be-d468-4c19-ae10-ec064e9a7af7" (UID: "b9c2f0be-d468-4c19-ae10-ec064e9a7af7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:42:08.687048 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.687020    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b9c2f0be-d468-4c19-ae10-ec064e9a7af7" (UID: "b9c2f0be-d468-4c19-ae10-ec064e9a7af7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:42:08.687306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.687090    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b9c2f0be-d468-4c19-ae10-ec064e9a7af7" (UID: "b9c2f0be-d468-4c19-ae10-ec064e9a7af7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:42:08.687306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.687238    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-data\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:08.687306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.687256    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istiod-ca-cert\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:08.687306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.687272    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-credential-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:08.687306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.687286    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-workload-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:08.687306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.687292    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b9c2f0be-d468-4c19-ae10-ec064e9a7af7" (UID: "b9c2f0be-d468-4c19-ae10-ec064e9a7af7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:42:08.689048 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.689016    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b9c2f0be-d468-4c19-ae10-ec064e9a7af7" (UID: "b9c2f0be-d468-4c19-ae10-ec064e9a7af7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:42:08.689166 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.689110    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-kube-api-access-7mwdc" (OuterVolumeSpecName: "kube-api-access-7mwdc") pod "b9c2f0be-d468-4c19-ae10-ec064e9a7af7" (UID: "b9c2f0be-d468-4c19-ae10-ec064e9a7af7"). InnerVolumeSpecName "kube-api-access-7mwdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:42:08.689236 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.689219    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-token" (OuterVolumeSpecName: "istio-token") pod "b9c2f0be-d468-4c19-ae10-ec064e9a7af7" (UID: "b9c2f0be-d468-4c19-ae10-ec064e9a7af7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:42:08.689297 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.689229    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b9c2f0be-d468-4c19-ae10-ec064e9a7af7" (UID: "b9c2f0be-d468-4c19-ae10-ec064e9a7af7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 19:42:08.788155 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.788104    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-podinfo\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:08.788155 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.788149    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-envoy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:08.788155 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.788158    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-istio-token\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:08.788155 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.788167    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mwdc\" (UniqueName: \"kubernetes.io/projected/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-kube-api-access-7mwdc\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:08.788155 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:08.788178    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c2f0be-d468-4c19-ae10-ec064e9a7af7-workload-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:09.115048 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:09.115012    2571 generic.go:356] "Generic (PLEG): container finished" podID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" containerID="91d7655eb2cda847524875fce79cac131734f37b939f3ca6d4d0192b32ffc5fe" exitCode=0
Sep 06 19:42:09.115216 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:09.115066    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"
Sep 06 19:42:09.115216 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:09.115101    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s" event={"ID":"b9c2f0be-d468-4c19-ae10-ec064e9a7af7","Type":"ContainerDied","Data":"91d7655eb2cda847524875fce79cac131734f37b939f3ca6d4d0192b32ffc5fe"}
Sep 06 19:42:09.115216 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:09.115156    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s" event={"ID":"b9c2f0be-d468-4c19-ae10-ec064e9a7af7","Type":"ContainerDied","Data":"20bcb0431f7b54e9c0680134c6b442693e69bdd3e0fcebdb0386a182093ee454"}
Sep 06 19:42:09.115216 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:09.115180    2571 scope.go:117] "RemoveContainer" containerID="91d7655eb2cda847524875fce79cac131734f37b939f3ca6d4d0192b32ffc5fe"
Sep 06 19:42:09.124922 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:09.124900    2571 scope.go:117] "RemoveContainer" containerID="03104b357845c7acc69fe5ae83bcae75f4ef19196d3863c491f20ff47460ba8f"
Sep 06 19:42:09.141980 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:09.141144    2571 scope.go:117] "RemoveContainer" containerID="91d7655eb2cda847524875fce79cac131734f37b939f3ca6d4d0192b32ffc5fe"
Sep 06 19:42:09.141980 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:42:09.141567    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d7655eb2cda847524875fce79cac131734f37b939f3ca6d4d0192b32ffc5fe\": container with ID starting with 91d7655eb2cda847524875fce79cac131734f37b939f3ca6d4d0192b32ffc5fe not found: ID does not exist" containerID="91d7655eb2cda847524875fce79cac131734f37b939f3ca6d4d0192b32ffc5fe"
Sep 06 19:42:09.141980 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:09.141601    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d7655eb2cda847524875fce79cac131734f37b939f3ca6d4d0192b32ffc5fe"} err="failed to get container status \"91d7655eb2cda847524875fce79cac131734f37b939f3ca6d4d0192b32ffc5fe\": rpc error: code = NotFound desc = could not find container \"91d7655eb2cda847524875fce79cac131734f37b939f3ca6d4d0192b32ffc5fe\": container with ID starting with 91d7655eb2cda847524875fce79cac131734f37b939f3ca6d4d0192b32ffc5fe not found: ID does not exist"
Sep 06 19:42:09.141980 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:09.141626    2571 scope.go:117] "RemoveContainer" containerID="03104b357845c7acc69fe5ae83bcae75f4ef19196d3863c491f20ff47460ba8f"
Sep 06 19:42:09.141980 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:42:09.141890    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03104b357845c7acc69fe5ae83bcae75f4ef19196d3863c491f20ff47460ba8f\": container with ID starting with 03104b357845c7acc69fe5ae83bcae75f4ef19196d3863c491f20ff47460ba8f not found: ID does not exist" containerID="03104b357845c7acc69fe5ae83bcae75f4ef19196d3863c491f20ff47460ba8f"
Sep 06 19:42:09.141980 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:09.141912    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03104b357845c7acc69fe5ae83bcae75f4ef19196d3863c491f20ff47460ba8f"} err="failed to get container status \"03104b357845c7acc69fe5ae83bcae75f4ef19196d3863c491f20ff47460ba8f\": rpc error: code = NotFound desc = could not find container \"03104b357845c7acc69fe5ae83bcae75f4ef19196d3863c491f20ff47460ba8f\": container with ID starting with 03104b357845c7acc69fe5ae83bcae75f4ef19196d3863c491f20ff47460ba8f not found: ID does not exist"
Sep 06 19:42:09.142467 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:09.142421    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"]
Sep 06 19:42:09.145238 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:09.145213    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4e3f8e3b-openshift-default-f6c85c8bd-lf72s"]
Sep 06 19:42:10.446996 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:10.446937    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:42:10.918663 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:10.918630    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" path="/var/lib/kubelet/pods/b9c2f0be-d468-4c19-ae10-ec064e9a7af7/volumes"
Sep 06 19:42:14.149179 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:14.149136    2571 generic.go:356] "Generic (PLEG): container finished" podID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" containerID="1ddd37391e7f4ec28a6ad89893f7e1a7fdeffbb7d75ba2dcb9ca887837a7e484" exitCode=0
Sep 06 19:42:14.149732 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:14.149260    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq" event={"ID":"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d","Type":"ContainerDied","Data":"1ddd37391e7f4ec28a6ad89893f7e1a7fdeffbb7d75ba2dcb9ca887837a7e484"}
Sep 06 19:42:14.149811 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:14.149777    2571 scope.go:117] "RemoveContainer" containerID="1ddd37391e7f4ec28a6ad89893f7e1a7fdeffbb7d75ba2dcb9ca887837a7e484"
Sep 06 19:42:14.151733 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:14.151687    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:42:14.151845 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:14.151791    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:42:14.151921 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:14.151842    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:42:15.161078 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:15.161036    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq" event={"ID":"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d","Type":"ContainerStarted","Data":"23c5c72b2fde08ed7b754bfa3ee7b4f6b8ba599b497c99dd5afd1c19480f2019"}
Sep 06 19:42:15.503792 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:15.503699    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:42:15.504085 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:15.504064    2571 patch_prober.go:28] interesting pod/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 06 19:42:15.504149 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:15.504108    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq" podUID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 06 19:42:16.503861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:16.503816    2571 patch_prober.go:28] interesting pod/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 06 19:42:16.504282 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:16.503873    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq" podUID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 06 19:42:16.915582 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:16.915556    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 19:42:16.916471 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:16.916453    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 19:42:17.504342 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:17.504298    2571 patch_prober.go:28] interesting pod/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 06 19:42:17.504742 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:17.504354    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq" podUID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 06 19:42:18.504448 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:18.504412    2571 patch_prober.go:28] interesting pod/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 06 19:42:18.505072 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:18.505031    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq" podUID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 06 19:42:19.504021 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:19.503983    2571 patch_prober.go:28] interesting pod/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 06 19:42:19.504205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:19.504041    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq" podUID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 06 19:42:20.600833 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:20.600796    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:42:20.601278 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:20.601193    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:42:20.602165 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:20.602148    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:42:21.285833 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:21.285696    2571 patch_prober.go:28] interesting pod/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": read tcp 10.132.0.2:36390->10.132.0.41:15021: read: connection reset by peer" start-of-body=
Sep 06 19:42:21.285833 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:21.285760    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8" podUID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": read tcp 10.132.0.2:36390->10.132.0.41:15021: read: connection reset by peer"
Sep 06 19:42:22.207585 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:22.207555    2571 generic.go:356] "Generic (PLEG): container finished" podID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" containerID="55cf443d6f5b2c7ca9bc385ed2dbc540490960350838ba96cdff6a6aeaacb7e8" exitCode=0
Sep 06 19:42:22.207585 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:22.207585    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8" event={"ID":"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f","Type":"ContainerDied","Data":"55cf443d6f5b2c7ca9bc385ed2dbc540490960350838ba96cdff6a6aeaacb7e8"}
Sep 06 19:42:22.208088 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:22.208060    2571 scope.go:117] "RemoveContainer" containerID="55cf443d6f5b2c7ca9bc385ed2dbc540490960350838ba96cdff6a6aeaacb7e8"
Sep 06 19:42:22.209365 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:22.209326    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:42:22.209465 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:22.209420    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:42:22.209510 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:22.209466    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:42:23.215000 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:23.214952    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8" event={"ID":"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f","Type":"ContainerStarted","Data":"141fba877dc5c69dc27783c4ff92215614d53972609b76fe92407a2a7f6a576e"}
Sep 06 19:42:23.546457 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:23.546360    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:23.546782 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:23.546760    2571 patch_prober.go:28] interesting pod/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Sep 06 19:42:23.546928 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:23.546903    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8" podUID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Sep 06 19:42:24.545906 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:24.545852    2571 patch_prober.go:28] interesting pod/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Sep 06 19:42:24.546445 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:24.545930    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8" podUID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Sep 06 19:42:25.546648 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:25.546609    2571 patch_prober.go:28] interesting pod/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Sep 06 19:42:25.547288 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:25.547255    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8" podUID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Sep 06 19:42:26.540976 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:26.540904    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:42:26.545923 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:26.545878    2571 patch_prober.go:28] interesting pod/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Sep 06 19:42:26.546135 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:26.545948    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8" podUID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Sep 06 19:42:27.545983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:27.545925    2571 patch_prober.go:28] interesting pod/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Sep 06 19:42:27.546583 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:27.546549    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8" podUID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Sep 06 19:42:28.550608 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:28.550573    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:28.551015 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:28.550877    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:28.551630 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:28.551613    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:31.433834 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:31.433793    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c"]
Sep 06 19:42:31.434345 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:31.434202    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" containerName="istio-proxy"
Sep 06 19:42:31.434345 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:31.434215    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" containerName="istio-proxy"
Sep 06 19:42:31.434345 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:31.434224    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" containerName="istio-proxy"
Sep 06 19:42:31.434345 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:31.434229    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" containerName="istio-proxy"
Sep 06 19:42:31.434345 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:31.434319    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" containerName="istio-proxy"
Sep 06 19:42:31.434345 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:31.434328    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9c2f0be-d468-4c19-ae10-ec064e9a7af7" containerName="istio-proxy"
Sep 06 19:42:31.437668 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:31.437652    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c"
Sep 06 19:42:31.440178 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:31.440154    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h45ww\""
Sep 06 19:42:31.440335 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:31.440154    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 19:42:31.444142 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:31.444116    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c"]
Sep 06 19:42:31.547866 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:31.547824    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b94629-bd1a-4640-b02f-20687f949fec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c\" (UID: \"60b94629-bd1a-4640-b02f-20687f949fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c"
Sep 06 19:42:31.649861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:31.649808    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b94629-bd1a-4640-b02f-20687f949fec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c\" (UID: \"60b94629-bd1a-4640-b02f-20687f949fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c"
Sep 06 19:42:31.650810 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:31.650774    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b94629-bd1a-4640-b02f-20687f949fec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c\" (UID: \"60b94629-bd1a-4640-b02f-20687f949fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c"
Sep 06 19:42:31.751062 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:31.750934    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c"
Sep 06 19:42:32.041938 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:32.041905    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c"]
Sep 06 19:42:32.043422 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:42:32.043387    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b94629_bd1a_4640_b02f_20687f949fec.slice/crio-6abce67e4f75bdf208e7b4d849616464077c142d9e07948f0a2673c9c8f8408a WatchSource:0}: Error finding container 6abce67e4f75bdf208e7b4d849616464077c142d9e07948f0a2673c9c8f8408a: Status 404 returned error can't find the container with id 6abce67e4f75bdf208e7b4d849616464077c142d9e07948f0a2673c9c8f8408a
Sep 06 19:42:32.265566 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:32.265523    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c" event={"ID":"60b94629-bd1a-4640-b02f-20687f949fec","Type":"ContainerStarted","Data":"6abce67e4f75bdf208e7b4d849616464077c142d9e07948f0a2673c9c8f8408a"}
Sep 06 19:42:33.272217 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:33.272181    2571 generic.go:356] "Generic (PLEG): container finished" podID="60b94629-bd1a-4640-b02f-20687f949fec" containerID="c1aef4a9a532b51c891abcd5ba68cdd78f400728c5084e1f316d7fefa4bca0c2" exitCode=0
Sep 06 19:42:33.272612 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:33.272231    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c" event={"ID":"60b94629-bd1a-4640-b02f-20687f949fec","Type":"ContainerDied","Data":"c1aef4a9a532b51c891abcd5ba68cdd78f400728c5084e1f316d7fefa4bca0c2"}
Sep 06 19:42:34.414539 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:34.414510    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c"
Sep 06 19:42:34.479876 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:34.479839    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b94629-bd1a-4640-b02f-20687f949fec-maas-api-service-ca\") pod \"60b94629-bd1a-4640-b02f-20687f949fec\" (UID: \"60b94629-bd1a-4640-b02f-20687f949fec\") "
Sep 06 19:42:34.480289 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:34.480266    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b94629-bd1a-4640-b02f-20687f949fec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60b94629-bd1a-4640-b02f-20687f949fec" (UID: "60b94629-bd1a-4640-b02f-20687f949fec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:42:34.581358 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:34.581315    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b94629-bd1a-4640-b02f-20687f949fec-maas-api-service-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:35.283915 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:35.283811    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c" event={"ID":"60b94629-bd1a-4640-b02f-20687f949fec","Type":"ContainerDied","Data":"6abce67e4f75bdf208e7b4d849616464077c142d9e07948f0a2673c9c8f8408a"}
Sep 06 19:42:35.283915 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:35.283867    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6abce67e4f75bdf208e7b4d849616464077c142d9e07948f0a2673c9c8f8408a"
Sep 06 19:42:35.283915 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:35.283821    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c"
Sep 06 19:42:38.095673 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:38.095630    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c"]
Sep 06 19:42:38.099609 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:38.099578    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4020e88-5s47c"]
Sep 06 19:42:38.218946 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:38.218904    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"]
Sep 06 19:42:38.219216 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:38.219187    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8" podUID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" containerName="istio-proxy" containerID="cri-o://141fba877dc5c69dc27783c4ff92215614d53972609b76fe92407a2a7f6a576e" gracePeriod=30
Sep 06 19:42:38.920740 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:38.920701    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60b94629-bd1a-4640-b02f-20687f949fec" path="/var/lib/kubelet/pods/60b94629-bd1a-4640-b02f-20687f949fec/volumes"
Sep 06 19:42:38.921031 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:38.921013    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"]
Sep 06 19:42:38.921444 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:38.921430    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b94629-bd1a-4640-b02f-20687f949fec" containerName="revoke-keys"
Sep 06 19:42:38.921505 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:38.921445    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b94629-bd1a-4640-b02f-20687f949fec" containerName="revoke-keys"
Sep 06 19:42:38.921565 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:38.921533    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b94629-bd1a-4640-b02f-20687f949fec" containerName="revoke-keys"
Sep 06 19:42:38.924780 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:38.924754    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:38.927211 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:38.927184    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b6954964-openshift-default-dockercfg-k2v7x\""
Sep 06 19:42:38.935330 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:38.935297    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"]
Sep 06 19:42:39.026395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.026357    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-data\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.026395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.026402    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-podinfo\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.026664 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.026550    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qj46\" (UniqueName: \"kubernetes.io/projected/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-kube-api-access-8qj46\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.026727 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.026673    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istiod-ca-cert\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.026727 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.026708    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-token\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.026855 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.026739    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-credential-socket\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.026855 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.026768    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-envoy\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.026855 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.026821    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-workload-certs\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.026855 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.026847    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-workload-socket\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.127744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.127702    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qj46\" (UniqueName: \"kubernetes.io/projected/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-kube-api-access-8qj46\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.128286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.127801    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istiod-ca-cert\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.128286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.127837    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-token\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.128286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.127871    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-credential-socket\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.128286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.127903    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-envoy\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.128286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.127945    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-workload-certs\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.128286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.128003    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-workload-socket\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.128286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.128039    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-data\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.128286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.128065    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-podinfo\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.128662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.128417    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-workload-socket\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.128662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.128442    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-credential-socket\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.128662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.128518    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-data\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.128662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.128546    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-workload-certs\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.128662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.128621    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istiod-ca-cert\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.130427 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.130402    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-envoy\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.130575 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.130556    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-podinfo\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.135289 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.135254    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-token\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.135394 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.135377    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qj46\" (UniqueName: \"kubernetes.io/projected/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-kube-api-access-8qj46\") pod \"e2e-ait-b6954964-openshift-default-5c7464db-l2xpp\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") " pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.237553 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.237451    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:39.516070 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.516023    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"]
Sep 06 19:42:39.519657 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:42:39.519271    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663e0201_eaf8_45f9_88a0_7e4abd6dbdc0.slice/crio-d1bb21079b971e7af98b98248ea54d27967f117fdd01cc06e1a518cc86e37fa0 WatchSource:0}: Error finding container d1bb21079b971e7af98b98248ea54d27967f117fdd01cc06e1a518cc86e37fa0: Status 404 returned error can't find the container with id d1bb21079b971e7af98b98248ea54d27967f117fdd01cc06e1a518cc86e37fa0
Sep 06 19:42:39.522369 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.522295    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:42:39.522500 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.522420    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:42:39.522500 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:39.522461    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:42:40.316674 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:40.316633    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp" event={"ID":"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0","Type":"ContainerStarted","Data":"10681c548786d2e842190a260c88b13b350f597f7627ed02f9b6eed47e5b90c0"}
Sep 06 19:42:40.317183 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:40.316716    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp" event={"ID":"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0","Type":"ContainerStarted","Data":"d1bb21079b971e7af98b98248ea54d27967f117fdd01cc06e1a518cc86e37fa0"}
Sep 06 19:42:40.334653 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:40.334599    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp" podStartSLOduration=2.334583832 podStartE2EDuration="2.334583832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:42:40.333370579 +0000 UTC m=+923.903668926" watchObservedRunningTime="2026-09-06 19:42:40.334583832 +0000 UTC m=+923.904882178"
Sep 06 19:42:41.238566 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:41.238530    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:41.243541 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:41.243516    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:41.322381 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:41.322346    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:41.323301 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:41.323278    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:42:41.932089 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:41.932048    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"]
Sep 06 19:42:41.935761 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:41.935744    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:41.938132 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:41.938114    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-6e7ee48c-kserve-self-signed-certs\""
Sep 06 19:42:41.944120 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:41.944085    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"]
Sep 06 19:42:42.060324 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.060286    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsggv\" (UniqueName: \"kubernetes.io/projected/807122d4-6fc3-4afb-b636-092cec85d529-kube-api-access-rsggv\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.060505 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.060350    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-home\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.060555 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.060529    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-dshm\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.060591 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.060560    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/807122d4-6fc3-4afb-b636-092cec85d529-tls-certs\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.060641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.060622    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-model-cache\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.060678 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.060662    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-tmp-dir\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.161596 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.161559    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-tmp-dir\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.161796 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.161649    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsggv\" (UniqueName: \"kubernetes.io/projected/807122d4-6fc3-4afb-b636-092cec85d529-kube-api-access-rsggv\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.161796 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.161698    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-home\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.161796 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.161744    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-dshm\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.161796 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.161758    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/807122d4-6fc3-4afb-b636-092cec85d529-tls-certs\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.161796 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.161781    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-model-cache\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.162092 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.162029    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-tmp-dir\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.162092 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.162078    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-model-cache\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.162207 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.162120    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-home\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.164102 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.164081    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-dshm\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.164359 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.164340    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/807122d4-6fc3-4afb-b636-092cec85d529-tls-certs\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.169376 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.169354    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsggv\" (UniqueName: \"kubernetes.io/projected/807122d4-6fc3-4afb-b636-092cec85d529-kube-api-access-rsggv\") pod \"e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") " pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.246731 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.246636    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:42.517589 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:42.517562    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"]
Sep 06 19:42:42.519541 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:42:42.519511    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod807122d4_6fc3_4afb_b636_092cec85d529.slice/crio-74d569a8be5c76b54f33bed2a14ea527cbe6fbb8e16d91ec0a311dcaef00dc23 WatchSource:0}: Error finding container 74d569a8be5c76b54f33bed2a14ea527cbe6fbb8e16d91ec0a311dcaef00dc23: Status 404 returned error can't find the container with id 74d569a8be5c76b54f33bed2a14ea527cbe6fbb8e16d91ec0a311dcaef00dc23
Sep 06 19:42:43.335480 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.335443    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb" event={"ID":"807122d4-6fc3-4afb-b636-092cec85d529","Type":"ContainerStarted","Data":"98a88bf59456e4fb10d355771414697d9450338b3228deb54933a1076638f2f8"}
Sep 06 19:42:43.335636 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.335493    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb" event={"ID":"807122d4-6fc3-4afb-b636-092cec85d529","Type":"ContainerStarted","Data":"74d569a8be5c76b54f33bed2a14ea527cbe6fbb8e16d91ec0a311dcaef00dc23"}
Sep 06 19:42:43.352289 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.352228    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb" podStartSLOduration=2.352208342 podStartE2EDuration="2.352208342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:42:43.350847103 +0000 UTC m=+926.921145449" watchObservedRunningTime="2026-09-06 19:42:43.352208342 +0000 UTC m=+926.922506716"
Sep 06 19:42:43.484436 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.484371    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:43.579408 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.579379    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istiod-ca-cert\") pod \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") "
Sep 06 19:42:43.579872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.579438    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-token\") pod \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") "
Sep 06 19:42:43.579872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.579469    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-envoy\") pod \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") "
Sep 06 19:42:43.579872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.579487    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-workload-certs\") pod \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") "
Sep 06 19:42:43.579872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.579518    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8fxh\" (UniqueName: \"kubernetes.io/projected/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-kube-api-access-n8fxh\") pod \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") "
Sep 06 19:42:43.579872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.579547    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-credential-socket\") pod \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") "
Sep 06 19:42:43.579872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.579568    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-workload-socket\") pod \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") "
Sep 06 19:42:43.579872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.579598    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-podinfo\") pod \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") "
Sep 06 19:42:43.579872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.579743    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-data\") pod \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\" (UID: \"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f\") "
Sep 06 19:42:43.579872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.579772    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" (UID: "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:42:43.579872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.579813    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" (UID: "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:42:43.579872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.579828    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" (UID: "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:42:43.580503 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.579934    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" (UID: "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:42:43.580503 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.580191    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-workload-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:43.580503 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.580219    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istiod-ca-cert\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:43.580503 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.580188    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-data" (OuterVolumeSpecName: "istio-data") pod "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" (UID: "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:42:43.580503 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.580233    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-workload-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:43.580503 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.580273    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-credential-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:43.581868 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.581839    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" (UID: "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:42:43.581868 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.581863    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-token" (OuterVolumeSpecName: "istio-token") pod "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" (UID: "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:42:43.582076 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.581842    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-kube-api-access-n8fxh" (OuterVolumeSpecName: "kube-api-access-n8fxh") pod "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" (UID: "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f"). InnerVolumeSpecName "kube-api-access-n8fxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:42:43.582076 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.581900    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" (UID: "a44f184f-8a05-4478-a03b-c2e1aaa0cb4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 19:42:43.680872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.680830    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-data\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:43.680872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.680865    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-token\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:43.680872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.680873    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-envoy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:43.680872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.680882    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8fxh\" (UniqueName: \"kubernetes.io/projected/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-kube-api-access-n8fxh\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:43.681183 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:43.680891    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f-istio-podinfo\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:42:44.350497 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:44.350464    2571 generic.go:356] "Generic (PLEG): container finished" podID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" containerID="141fba877dc5c69dc27783c4ff92215614d53972609b76fe92407a2a7f6a576e" exitCode=0
Sep 06 19:42:44.350682 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:44.350521    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"
Sep 06 19:42:44.350682 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:44.350545    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8" event={"ID":"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f","Type":"ContainerDied","Data":"141fba877dc5c69dc27783c4ff92215614d53972609b76fe92407a2a7f6a576e"}
Sep 06 19:42:44.350682 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:44.350618    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8" event={"ID":"a44f184f-8a05-4478-a03b-c2e1aaa0cb4f","Type":"ContainerDied","Data":"9e371a23ef10403aa01d9da50542367ad467eca6ba3046a72dbb29691d9c0b54"}
Sep 06 19:42:44.350682 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:44.350640    2571 scope.go:117] "RemoveContainer" containerID="141fba877dc5c69dc27783c4ff92215614d53972609b76fe92407a2a7f6a576e"
Sep 06 19:42:44.360788 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:44.360765    2571 scope.go:117] "RemoveContainer" containerID="55cf443d6f5b2c7ca9bc385ed2dbc540490960350838ba96cdff6a6aeaacb7e8"
Sep 06 19:42:44.369545 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:44.369449    2571 scope.go:117] "RemoveContainer" containerID="141fba877dc5c69dc27783c4ff92215614d53972609b76fe92407a2a7f6a576e"
Sep 06 19:42:44.369828 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:42:44.369804    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141fba877dc5c69dc27783c4ff92215614d53972609b76fe92407a2a7f6a576e\": container with ID starting with 141fba877dc5c69dc27783c4ff92215614d53972609b76fe92407a2a7f6a576e not found: ID does not exist" containerID="141fba877dc5c69dc27783c4ff92215614d53972609b76fe92407a2a7f6a576e"
Sep 06 19:42:44.369897 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:44.369837    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141fba877dc5c69dc27783c4ff92215614d53972609b76fe92407a2a7f6a576e"} err="failed to get container status \"141fba877dc5c69dc27783c4ff92215614d53972609b76fe92407a2a7f6a576e\": rpc error: code = NotFound desc = could not find container \"141fba877dc5c69dc27783c4ff92215614d53972609b76fe92407a2a7f6a576e\": container with ID starting with 141fba877dc5c69dc27783c4ff92215614d53972609b76fe92407a2a7f6a576e not found: ID does not exist"
Sep 06 19:42:44.369897 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:44.369856    2571 scope.go:117] "RemoveContainer" containerID="55cf443d6f5b2c7ca9bc385ed2dbc540490960350838ba96cdff6a6aeaacb7e8"
Sep 06 19:42:44.370127 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:42:44.370084    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55cf443d6f5b2c7ca9bc385ed2dbc540490960350838ba96cdff6a6aeaacb7e8\": container with ID starting with 55cf443d6f5b2c7ca9bc385ed2dbc540490960350838ba96cdff6a6aeaacb7e8 not found: ID does not exist" containerID="55cf443d6f5b2c7ca9bc385ed2dbc540490960350838ba96cdff6a6aeaacb7e8"
Sep 06 19:42:44.370127 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:44.370110    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55cf443d6f5b2c7ca9bc385ed2dbc540490960350838ba96cdff6a6aeaacb7e8"} err="failed to get container status \"55cf443d6f5b2c7ca9bc385ed2dbc540490960350838ba96cdff6a6aeaacb7e8\": rpc error: code = NotFound desc = could not find container \"55cf443d6f5b2c7ca9bc385ed2dbc540490960350838ba96cdff6a6aeaacb7e8\": container with ID starting with 55cf443d6f5b2c7ca9bc385ed2dbc540490960350838ba96cdff6a6aeaacb7e8 not found: ID does not exist"
Sep 06 19:42:44.371275 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:44.371250    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"]
Sep 06 19:42:44.374590 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:44.374567    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b4020e88-gw-openshift-default-85f5798dc6-krpt8"]
Sep 06 19:42:44.919018 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:44.918987    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" path="/var/lib/kubelet/pods/a44f184f-8a05-4478-a03b-c2e1aaa0cb4f/volumes"
Sep 06 19:42:45.750207 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:45.750173    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:42:52.247775 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:52.247736    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:52.248240 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:52.247794    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:52.260610 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:52.260580    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:42:52.407299 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:42:52.407269    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:43:00.741517 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.741478    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"]
Sep 06 19:43:00.747982 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.742665    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" containerName="istio-proxy"
Sep 06 19:43:00.751885 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.751857    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" containerName="istio-proxy"
Sep 06 19:43:00.752338 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.752320    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" containerName="istio-proxy"
Sep 06 19:43:00.752491 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.752478    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" containerName="istio-proxy"
Sep 06 19:43:00.768323 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.768282    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"]
Sep 06 19:43:00.768476 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.768415    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.772071 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.772033    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0ebbc9\"/\"default-dockercfg-d84s4\""
Sep 06 19:43:00.772214 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.772089    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0ebbc9\"/\"autores-model-0ebbc9-kserve-self-signed-certs\""
Sep 06 19:43:00.772214 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.772089    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0ebbc9\"/\"openshift-service-ca.crt\""
Sep 06 19:43:00.772371 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.772237    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0ebbc9\"/\"kube-root-ca.crt\""
Sep 06 19:43:00.864659 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.864609    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-dshm\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.864659 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.864665    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqf57\" (UniqueName: \"kubernetes.io/projected/026ef435-1597-441a-a300-fcc8873f8c5d-kube-api-access-wqf57\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.864927 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.864779    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-tmp-dir\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.864927 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.864814    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026ef435-1597-441a-a300-fcc8873f8c5d-tls-certs\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.864927 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.864911    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-home\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.865104 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.864947    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-model-cache\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.966008 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.965946    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-tmp-dir\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.966224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.966026    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026ef435-1597-441a-a300-fcc8873f8c5d-tls-certs\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.966224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.966091    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-home\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.966224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.966119    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-model-cache\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.966408 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.966225    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-dshm\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.966408 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.966267    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqf57\" (UniqueName: \"kubernetes.io/projected/026ef435-1597-441a-a300-fcc8873f8c5d-kube-api-access-wqf57\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.967007 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.966980    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-tmp-dir\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.967260 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.967236    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-model-cache\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.967625 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.967605    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-home\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.969989 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.969952    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026ef435-1597-441a-a300-fcc8873f8c5d-tls-certs\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.971981 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.970798    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-dshm\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:00.973564 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:00.973530    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqf57\" (UniqueName: \"kubernetes.io/projected/026ef435-1597-441a-a300-fcc8873f8c5d-kube-api-access-wqf57\") pod \"autores-model-0ebbc9-kserve-665c6cbbdc-ls44t\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") " pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:01.078970 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:01.078931    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:01.351043 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:01.350923    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"]
Sep 06 19:43:01.472792 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:01.472759    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t" event={"ID":"026ef435-1597-441a-a300-fcc8873f8c5d","Type":"ContainerStarted","Data":"c4e219b014fcf2c095e12112b8f9b7bc60c9c04efa1acc83d00776152874c415"}
Sep 06 19:43:01.473012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:01.472801    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t" event={"ID":"026ef435-1597-441a-a300-fcc8873f8c5d","Type":"ContainerStarted","Data":"1f15bd7f7b3988ec7047ba4650c75baaf4fcd4699aaf2d9b997d81d279739a13"}
Sep 06 19:43:01.487508 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:01.487450    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t" podStartSLOduration=1.4874312619999999 podStartE2EDuration="1.487431262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:43:01.48637008 +0000 UTC m=+945.056668426" watchObservedRunningTime="2026-09-06 19:43:01.487431262 +0000 UTC m=+945.057729639"
Sep 06 19:43:03.534747 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:03.534705    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:43:06.503421 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:06.503387    2571 generic.go:356] "Generic (PLEG): container finished" podID="663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" containerID="10681c548786d2e842190a260c88b13b350f597f7627ed02f9b6eed47e5b90c0" exitCode=0
Sep 06 19:43:06.503913 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:06.503433    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp" event={"ID":"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0","Type":"ContainerDied","Data":"10681c548786d2e842190a260c88b13b350f597f7627ed02f9b6eed47e5b90c0"}
Sep 06 19:43:06.503913 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:06.503853    2571 scope.go:117] "RemoveContainer" containerID="10681c548786d2e842190a260c88b13b350f597f7627ed02f9b6eed47e5b90c0"
Sep 06 19:43:06.505015 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:06.504983    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:43:06.505097 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:06.505053    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:43:06.505137 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:06.505105    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:43:07.291651 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:07.291606    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9"]
Sep 06 19:43:07.292126 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:07.292106    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" containerName="istio-proxy"
Sep 06 19:43:07.292126 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:07.292128    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a44f184f-8a05-4478-a03b-c2e1aaa0cb4f" containerName="istio-proxy"
Sep 06 19:43:07.294251 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:07.294227    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9"
Sep 06 19:43:07.296800 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:07.296767    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 19:43:07.296930 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:07.296825    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h45ww\""
Sep 06 19:43:07.302899 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:07.302869    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9"]
Sep 06 19:43:07.434409 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:07.434311    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16826fb7-dcab-4056-a263-4f0a73a8a707-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b6954964-hcql9\" (UID: \"16826fb7-dcab-4056-a263-4f0a73a8a707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9"
Sep 06 19:43:07.511026 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:07.510982    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp" event={"ID":"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0","Type":"ContainerStarted","Data":"67f2444a62f7b957502ca2daa9d6a4e4c93f35b218970c64bc0e76a415ca95f6"}
Sep 06 19:43:07.536011 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:07.535948    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16826fb7-dcab-4056-a263-4f0a73a8a707-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b6954964-hcql9\" (UID: \"16826fb7-dcab-4056-a263-4f0a73a8a707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9"
Sep 06 19:43:07.536762 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:07.536731    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16826fb7-dcab-4056-a263-4f0a73a8a707-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b6954964-hcql9\" (UID: \"16826fb7-dcab-4056-a263-4f0a73a8a707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9"
Sep 06 19:43:07.606280 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:07.606243    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9"
Sep 06 19:43:07.923094 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:07.923064    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9"]
Sep 06 19:43:07.924639 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:43:07.924615    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16826fb7_dcab_4056_a263_4f0a73a8a707.slice/crio-c67783cc4025ec1983a9c82edda788c048b23c0b5c23856609b647572bf2b13a WatchSource:0}: Error finding container c67783cc4025ec1983a9c82edda788c048b23c0b5c23856609b647572bf2b13a: Status 404 returned error can't find the container with id c67783cc4025ec1983a9c82edda788c048b23c0b5c23856609b647572bf2b13a
Sep 06 19:43:08.237668 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:08.237564    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:43:08.239020 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:08.238984    2571 patch_prober.go:28] interesting pod/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 06 19:43:08.239170 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:08.239062    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp" podUID="663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 06 19:43:08.517455 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:08.517354    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9" event={"ID":"16826fb7-dcab-4056-a263-4f0a73a8a707","Type":"ContainerStarted","Data":"430492765ce139275f138a43c6ad8feff017ab90ae3becdfbb65ae9bf4aeb917"}
Sep 06 19:43:08.517455 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:08.517421    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9" event={"ID":"16826fb7-dcab-4056-a263-4f0a73a8a707","Type":"ContainerStarted","Data":"c67783cc4025ec1983a9c82edda788c048b23c0b5c23856609b647572bf2b13a"}
Sep 06 19:43:08.532253 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:08.532196    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9" podStartSLOduration=1.53217498 podStartE2EDuration="1.53217498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:43:08.530244178 +0000 UTC m=+952.100542526" watchObservedRunningTime="2026-09-06 19:43:08.53217498 +0000 UTC m=+952.102473326"
Sep 06 19:43:09.238272 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:09.238162    2571 patch_prober.go:28] interesting pod/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 06 19:43:09.238272 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:09.238236    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp" podUID="663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 06 19:43:09.524024 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:09.523918    2571 generic.go:356] "Generic (PLEG): container finished" podID="16826fb7-dcab-4056-a263-4f0a73a8a707" containerID="430492765ce139275f138a43c6ad8feff017ab90ae3becdfbb65ae9bf4aeb917" exitCode=0
Sep 06 19:43:09.524024 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:09.524014    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9" event={"ID":"16826fb7-dcab-4056-a263-4f0a73a8a707","Type":"ContainerDied","Data":"430492765ce139275f138a43c6ad8feff017ab90ae3becdfbb65ae9bf4aeb917"}
Sep 06 19:43:10.238928 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:10.238887    2571 patch_prober.go:28] interesting pod/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 06 19:43:10.239249 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:10.239219    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp" podUID="663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 06 19:43:10.706144 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:10.706113    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9"
Sep 06 19:43:10.789180 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:10.789092    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16826fb7-dcab-4056-a263-4f0a73a8a707-maas-api-service-ca\") pod \"16826fb7-dcab-4056-a263-4f0a73a8a707\" (UID: \"16826fb7-dcab-4056-a263-4f0a73a8a707\") "
Sep 06 19:43:10.789619 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:10.789584    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16826fb7-dcab-4056-a263-4f0a73a8a707-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "16826fb7-dcab-4056-a263-4f0a73a8a707" (UID: "16826fb7-dcab-4056-a263-4f0a73a8a707"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:43:10.890806 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:10.890769    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16826fb7-dcab-4056-a263-4f0a73a8a707-maas-api-service-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:11.079630 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:11.079578    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:11.079911 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:11.079878    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:11.091986 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:11.091948    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:11.238823 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:11.238371    2571 patch_prober.go:28] interesting pod/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 06 19:43:11.238823 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:11.238435    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp" podUID="663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 06 19:43:11.322932 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:11.322895    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:43:11.540746 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:11.540649    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9"
Sep 06 19:43:11.541200 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:11.541175    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9" event={"ID":"16826fb7-dcab-4056-a263-4f0a73a8a707","Type":"ContainerDied","Data":"c67783cc4025ec1983a9c82edda788c048b23c0b5c23856609b647572bf2b13a"}
Sep 06 19:43:11.541324 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:11.541223    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c67783cc4025ec1983a9c82edda788c048b23c0b5c23856609b647572bf2b13a"
Sep 06 19:43:11.570498 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:11.570467    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:12.238297 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:12.238255    2571 patch_prober.go:28] interesting pod/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 06 19:43:12.238878 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:12.238342    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp" podUID="663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 06 19:43:13.243297 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:13.243266    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:43:13.244591 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:13.244566    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:43:13.779833 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:13.779794    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9"]
Sep 06 19:43:13.781654 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:13.781631    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b6954964-hcql9"]
Sep 06 19:43:14.861043 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:14.861002    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"]
Sep 06 19:43:14.861529 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:14.861244    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp" podUID="663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" containerName="istio-proxy" containerID="cri-o://67f2444a62f7b957502ca2daa9d6a4e4c93f35b218970c64bc0e76a415ca95f6" gracePeriod=30
Sep 06 19:43:14.918557 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:14.918520    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16826fb7-dcab-4056-a263-4f0a73a8a707" path="/var/lib/kubelet/pods/16826fb7-dcab-4056-a263-4f0a73a8a707/volumes"
Sep 06 19:43:19.852566 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:19.852515    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:43:20.126661 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.126633    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:43:20.188852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.188815    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-envoy\") pod \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") "
Sep 06 19:43:20.188852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.188858    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istiod-ca-cert\") pod \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") "
Sep 06 19:43:20.189134 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.188887    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-workload-certs\") pod \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") "
Sep 06 19:43:20.189134 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.188925    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-credential-socket\") pod \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") "
Sep 06 19:43:20.189134 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.188944    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-workload-socket\") pod \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") "
Sep 06 19:43:20.189134 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.189026    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-token\") pod \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") "
Sep 06 19:43:20.189134 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.189088    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-data\") pod \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") "
Sep 06 19:43:20.189375 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.189139    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-podinfo\") pod \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") "
Sep 06 19:43:20.189375 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.189182    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qj46\" (UniqueName: \"kubernetes.io/projected/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-kube-api-access-8qj46\") pod \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\" (UID: \"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0\") "
Sep 06 19:43:20.189375 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.189235    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" (UID: "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:20.189375 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.189257    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" (UID: "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:20.189375 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.189356    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" (UID: "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:20.189611 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.189374    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" (UID: "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:43:20.189611 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.189581    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-data" (OuterVolumeSpecName: "istio-data") pod "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" (UID: "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:20.189728 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.189660    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istiod-ca-cert\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:20.189728 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.189679    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-workload-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:20.189728 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.189687    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-credential-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:20.189728 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.189696    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-workload-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:20.189728 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.189705    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-data\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:20.191434 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.191406    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-kube-api-access-8qj46" (OuterVolumeSpecName: "kube-api-access-8qj46") pod "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" (UID: "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0"). InnerVolumeSpecName "kube-api-access-8qj46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:43:20.191536 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.191412    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-token" (OuterVolumeSpecName: "istio-token") pod "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" (UID: "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:43:20.191596 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.191581    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" (UID: "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:20.191659 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.191636    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" (UID: "663e0201-eaf8-45f9-88a0-7e4abd6dbdc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 19:43:20.290549 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.290504    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-token\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:20.290549 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.290540    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-podinfo\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:20.290549 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.290550    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qj46\" (UniqueName: \"kubernetes.io/projected/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-kube-api-access-8qj46\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:20.290549 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.290558    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0-istio-envoy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:20.591887 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.591852    2571 generic.go:356] "Generic (PLEG): container finished" podID="663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" containerID="67f2444a62f7b957502ca2daa9d6a4e4c93f35b218970c64bc0e76a415ca95f6" exitCode=0
Sep 06 19:43:20.592093 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.591895    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp" event={"ID":"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0","Type":"ContainerDied","Data":"67f2444a62f7b957502ca2daa9d6a4e4c93f35b218970c64bc0e76a415ca95f6"}
Sep 06 19:43:20.592093 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.591943    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp" event={"ID":"663e0201-eaf8-45f9-88a0-7e4abd6dbdc0","Type":"ContainerDied","Data":"d1bb21079b971e7af98b98248ea54d27967f117fdd01cc06e1a518cc86e37fa0"}
Sep 06 19:43:20.592093 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.591908    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"
Sep 06 19:43:20.592093 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.591998    2571 scope.go:117] "RemoveContainer" containerID="67f2444a62f7b957502ca2daa9d6a4e4c93f35b218970c64bc0e76a415ca95f6"
Sep 06 19:43:20.603735 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.603692    2571 scope.go:117] "RemoveContainer" containerID="10681c548786d2e842190a260c88b13b350f597f7627ed02f9b6eed47e5b90c0"
Sep 06 19:43:20.612124 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.612100    2571 scope.go:117] "RemoveContainer" containerID="67f2444a62f7b957502ca2daa9d6a4e4c93f35b218970c64bc0e76a415ca95f6"
Sep 06 19:43:20.612437 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:43:20.612418    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f2444a62f7b957502ca2daa9d6a4e4c93f35b218970c64bc0e76a415ca95f6\": container with ID starting with 67f2444a62f7b957502ca2daa9d6a4e4c93f35b218970c64bc0e76a415ca95f6 not found: ID does not exist" containerID="67f2444a62f7b957502ca2daa9d6a4e4c93f35b218970c64bc0e76a415ca95f6"
Sep 06 19:43:20.612501 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.612445    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f2444a62f7b957502ca2daa9d6a4e4c93f35b218970c64bc0e76a415ca95f6"} err="failed to get container status \"67f2444a62f7b957502ca2daa9d6a4e4c93f35b218970c64bc0e76a415ca95f6\": rpc error: code = NotFound desc = could not find container \"67f2444a62f7b957502ca2daa9d6a4e4c93f35b218970c64bc0e76a415ca95f6\": container with ID starting with 67f2444a62f7b957502ca2daa9d6a4e4c93f35b218970c64bc0e76a415ca95f6 not found: ID does not exist"
Sep 06 19:43:20.612501 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.612463    2571 scope.go:117] "RemoveContainer" containerID="10681c548786d2e842190a260c88b13b350f597f7627ed02f9b6eed47e5b90c0"
Sep 06 19:43:20.612781 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:43:20.612752    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10681c548786d2e842190a260c88b13b350f597f7627ed02f9b6eed47e5b90c0\": container with ID starting with 10681c548786d2e842190a260c88b13b350f597f7627ed02f9b6eed47e5b90c0 not found: ID does not exist" containerID="10681c548786d2e842190a260c88b13b350f597f7627ed02f9b6eed47e5b90c0"
Sep 06 19:43:20.612864 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.612788    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10681c548786d2e842190a260c88b13b350f597f7627ed02f9b6eed47e5b90c0"} err="failed to get container status \"10681c548786d2e842190a260c88b13b350f597f7627ed02f9b6eed47e5b90c0\": rpc error: code = NotFound desc = could not find container \"10681c548786d2e842190a260c88b13b350f597f7627ed02f9b6eed47e5b90c0\": container with ID starting with 10681c548786d2e842190a260c88b13b350f597f7627ed02f9b6eed47e5b90c0 not found: ID does not exist"
Sep 06 19:43:20.615246 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.615222    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"]
Sep 06 19:43:20.622835 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.622811    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b6954964-openshift-default-5c7464db-l2xpp"]
Sep 06 19:43:20.919943 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:20.919840    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" path="/var/lib/kubelet/pods/663e0201-eaf8-45f9-88a0-7e4abd6dbdc0/volumes"
Sep 06 19:43:23.226340 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.226299    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"]
Sep 06 19:43:23.226742 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.226696    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16826fb7-dcab-4056-a263-4f0a73a8a707" containerName="revoke-keys"
Sep 06 19:43:23.226742 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.226707    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="16826fb7-dcab-4056-a263-4f0a73a8a707" containerName="revoke-keys"
Sep 06 19:43:23.226742 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.226732    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" containerName="istio-proxy"
Sep 06 19:43:23.226742 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.226737    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" containerName="istio-proxy"
Sep 06 19:43:23.226873 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.226745    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" containerName="istio-proxy"
Sep 06 19:43:23.226873 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.226751    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" containerName="istio-proxy"
Sep 06 19:43:23.226873 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.226807    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" containerName="istio-proxy"
Sep 06 19:43:23.226873 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.226814    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="16826fb7-dcab-4056-a263-4f0a73a8a707" containerName="revoke-keys"
Sep 06 19:43:23.231419 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.231400    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.233805 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.233776    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6433a6\"/\"kube-root-ca.crt\""
Sep 06 19:43:23.234929 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.234899    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6433a6\"/\"test-model-6433a6-kserve-self-signed-certs\""
Sep 06 19:43:23.235094 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.234978    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6433a6\"/\"default-dockercfg-79wgw\""
Sep 06 19:43:23.235417 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.235387    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6433a6\"/\"openshift-service-ca.crt\""
Sep 06 19:43:23.238054 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.238028    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"]
Sep 06 19:43:23.320479 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.320437    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-dshm\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.320658 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.320484    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s47fv\" (UniqueName: \"kubernetes.io/projected/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-kube-api-access-s47fv\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.320658 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.320570    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-tls-certs\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.320658 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.320598    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-model-cache\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.320658 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.320648    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-home\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.320820 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.320694    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-tmp-dir\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.421838 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.421785    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-tls-certs\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.421838 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.421828    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-model-cache\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.422123 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.421891    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-home\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.422123 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.422013    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-tmp-dir\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.422243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.422124    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-dshm\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.422243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.422167    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s47fv\" (UniqueName: \"kubernetes.io/projected/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-kube-api-access-s47fv\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.422354 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.422315    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-home\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.422409 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.422355    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-model-cache\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.422409 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.422391    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-tmp-dir\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.424258 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.424227    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-dshm\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.424505 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.424486    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-tls-certs\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.429772 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.429750    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s47fv\" (UniqueName: \"kubernetes.io/projected/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-kube-api-access-s47fv\") pod \"test-model-6433a6-kserve-656666fd6c-lpttc\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") " pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.542298 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.542193    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:23.811123 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:23.811094    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"]
Sep 06 19:43:23.813271 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:43:23.813243    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode322d7ba_9018_40e9_914d_8e4ca2d6ca0e.slice/crio-89f6deb2bd24dd9ec5f285f2b887101b488d2934462cdd869cd97466e43667b3 WatchSource:0}: Error finding container 89f6deb2bd24dd9ec5f285f2b887101b488d2934462cdd869cd97466e43667b3: Status 404 returned error can't find the container with id 89f6deb2bd24dd9ec5f285f2b887101b488d2934462cdd869cd97466e43667b3
Sep 06 19:43:24.621718 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:24.621676    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc" event={"ID":"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e","Type":"ContainerStarted","Data":"714b41238b5511e46e7a462f5aa13305654d50a56bd2472c15b8b6cfc32ddabf"}
Sep 06 19:43:24.622141 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:24.621734    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc" event={"ID":"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e","Type":"ContainerStarted","Data":"89f6deb2bd24dd9ec5f285f2b887101b488d2934462cdd869cd97466e43667b3"}
Sep 06 19:43:24.637045 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:24.636988    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc" podStartSLOduration=1.6369538989999999 podStartE2EDuration="1.636953899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:43:24.636620798 +0000 UTC m=+968.206919145" watchObservedRunningTime="2026-09-06 19:43:24.636953899 +0000 UTC m=+968.207252246"
Sep 06 19:43:27.219596 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:27.219555    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"]
Sep 06 19:43:29.233770 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:29.233735    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"]
Sep 06 19:43:32.256234 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:32.256189    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"]
Sep 06 19:43:32.367641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:32.367602    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj"]
Sep 06 19:43:32.368150 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:32.368131    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="663e0201-eaf8-45f9-88a0-7e4abd6dbdc0" containerName="istio-proxy"
Sep 06 19:43:32.371384 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:32.371361    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj"
Sep 06 19:43:32.373808 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:32.373784    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 19:43:32.373930 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:32.373784    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h45ww\""
Sep 06 19:43:32.378487 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:32.378462    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj"]
Sep 06 19:43:32.521625 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:32.521539    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/654441ed-52e4-4ced-87ee-f9e84f662271-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj\" (UID: \"654441ed-52e4-4ced-87ee-f9e84f662271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj"
Sep 06 19:43:32.622729 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:32.622690    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/654441ed-52e4-4ced-87ee-f9e84f662271-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj\" (UID: \"654441ed-52e4-4ced-87ee-f9e84f662271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj"
Sep 06 19:43:32.623553 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:32.623526    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/654441ed-52e4-4ced-87ee-f9e84f662271-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj\" (UID: \"654441ed-52e4-4ced-87ee-f9e84f662271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj"
Sep 06 19:43:32.687590 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:32.687539    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj"
Sep 06 19:43:32.969382 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:32.969353    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj"]
Sep 06 19:43:32.972576 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:43:32.972539    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654441ed_52e4_4ced_87ee_f9e84f662271.slice/crio-45936f9d1b499c5d8e2822a97f14636d08c5e5c6df8c2843e447ce136ee0602c WatchSource:0}: Error finding container 45936f9d1b499c5d8e2822a97f14636d08c5e5c6df8c2843e447ce136ee0602c: Status 404 returned error can't find the container with id 45936f9d1b499c5d8e2822a97f14636d08c5e5c6df8c2843e447ce136ee0602c
Sep 06 19:43:33.542589 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:33.542491    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:33.542589 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:33.542573    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:33.555185 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:33.555157    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:33.668837 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:33.668799    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj" event={"ID":"654441ed-52e4-4ced-87ee-f9e84f662271","Type":"ContainerStarted","Data":"a57c065a26948d3e47454186ece45cb98b33104c609595675785a6348faa4517"}
Sep 06 19:43:33.669015 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:33.668856    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj" event={"ID":"654441ed-52e4-4ced-87ee-f9e84f662271","Type":"ContainerStarted","Data":"45936f9d1b499c5d8e2822a97f14636d08c5e5c6df8c2843e447ce136ee0602c"}
Sep 06 19:43:33.680394 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:33.680367    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:43:33.683797 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:33.683750    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj" podStartSLOduration=1.683735837 podStartE2EDuration="1.683735837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:43:33.682664624 +0000 UTC m=+977.252962970" watchObservedRunningTime="2026-09-06 19:43:33.683735837 +0000 UTC m=+977.254034215"
Sep 06 19:43:34.675844 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:34.675810    2571 generic.go:356] "Generic (PLEG): container finished" podID="654441ed-52e4-4ced-87ee-f9e84f662271" containerID="a57c065a26948d3e47454186ece45cb98b33104c609595675785a6348faa4517" exitCode=0
Sep 06 19:43:34.676264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:34.675898    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj" event={"ID":"654441ed-52e4-4ced-87ee-f9e84f662271","Type":"ContainerDied","Data":"a57c065a26948d3e47454186ece45cb98b33104c609595675785a6348faa4517"}
Sep 06 19:43:35.824608 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:35.824578    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj"
Sep 06 19:43:35.958753 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:35.958655    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/654441ed-52e4-4ced-87ee-f9e84f662271-maas-api-service-ca\") pod \"654441ed-52e4-4ced-87ee-f9e84f662271\" (UID: \"654441ed-52e4-4ced-87ee-f9e84f662271\") "
Sep 06 19:43:35.959045 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:35.959022    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654441ed-52e4-4ced-87ee-f9e84f662271-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "654441ed-52e4-4ced-87ee-f9e84f662271" (UID: "654441ed-52e4-4ced-87ee-f9e84f662271"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:43:36.060262 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:36.060222    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/654441ed-52e4-4ced-87ee-f9e84f662271-maas-api-service-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:36.688349 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:36.688323    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj"
Sep 06 19:43:36.688349 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:36.688343    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj" event={"ID":"654441ed-52e4-4ced-87ee-f9e84f662271","Type":"ContainerDied","Data":"45936f9d1b499c5d8e2822a97f14636d08c5e5c6df8c2843e447ce136ee0602c"}
Sep 06 19:43:36.688556 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:36.688383    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45936f9d1b499c5d8e2822a97f14636d08c5e5c6df8c2843e447ce136ee0602c"
Sep 06 19:43:42.240482 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.240382    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc" podUID="afa9ed92-6362-422e-9ff1-6f93523ffc01" containerName="main" containerID="cri-o://84ade560a206976191fdb7cdc0accaa829ca39440df8831dff984cb40fd929ca" gracePeriod=45
Sep 06 19:43:42.340132 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.340094    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:43:42.494310 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.494244    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:43:42.624294 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.624260    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-model-cache\") pod \"afa9ed92-6362-422e-9ff1-6f93523ffc01\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") "
Sep 06 19:43:42.624294 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.624300    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-tmp-dir\") pod \"afa9ed92-6362-422e-9ff1-6f93523ffc01\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") "
Sep 06 19:43:42.624550 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.624361    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgqsx\" (UniqueName: \"kubernetes.io/projected/afa9ed92-6362-422e-9ff1-6f93523ffc01-kube-api-access-hgqsx\") pod \"afa9ed92-6362-422e-9ff1-6f93523ffc01\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") "
Sep 06 19:43:42.624550 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.624395    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa9ed92-6362-422e-9ff1-6f93523ffc01-tls-certs\") pod \"afa9ed92-6362-422e-9ff1-6f93523ffc01\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") "
Sep 06 19:43:42.624550 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.624478    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-dshm\") pod \"afa9ed92-6362-422e-9ff1-6f93523ffc01\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") "
Sep 06 19:43:42.624550 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.624507    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-home\") pod \"afa9ed92-6362-422e-9ff1-6f93523ffc01\" (UID: \"afa9ed92-6362-422e-9ff1-6f93523ffc01\") "
Sep 06 19:43:42.624761 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.624585    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-model-cache" (OuterVolumeSpecName: "model-cache") pod "afa9ed92-6362-422e-9ff1-6f93523ffc01" (UID: "afa9ed92-6362-422e-9ff1-6f93523ffc01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:42.624761 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.624662    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "afa9ed92-6362-422e-9ff1-6f93523ffc01" (UID: "afa9ed92-6362-422e-9ff1-6f93523ffc01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:42.624856 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.624825    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-home" (OuterVolumeSpecName: "home") pod "afa9ed92-6362-422e-9ff1-6f93523ffc01" (UID: "afa9ed92-6362-422e-9ff1-6f93523ffc01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:42.624995 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.624979    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-home\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:42.625064 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.625002    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-model-cache\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:42.625064 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.625017    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-tmp-dir\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:42.626859 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.626833    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa9ed92-6362-422e-9ff1-6f93523ffc01-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "afa9ed92-6362-422e-9ff1-6f93523ffc01" (UID: "afa9ed92-6362-422e-9ff1-6f93523ffc01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:43:42.626859 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.626845    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa9ed92-6362-422e-9ff1-6f93523ffc01-kube-api-access-hgqsx" (OuterVolumeSpecName: "kube-api-access-hgqsx") pod "afa9ed92-6362-422e-9ff1-6f93523ffc01" (UID: "afa9ed92-6362-422e-9ff1-6f93523ffc01"). InnerVolumeSpecName "kube-api-access-hgqsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:43:42.627013 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.626888    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-dshm" (OuterVolumeSpecName: "dshm") pod "afa9ed92-6362-422e-9ff1-6f93523ffc01" (UID: "afa9ed92-6362-422e-9ff1-6f93523ffc01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:42.726015 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.725979    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgqsx\" (UniqueName: \"kubernetes.io/projected/afa9ed92-6362-422e-9ff1-6f93523ffc01-kube-api-access-hgqsx\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:42.726274 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.726248    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa9ed92-6362-422e-9ff1-6f93523ffc01-tls-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:42.726354 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.726287    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afa9ed92-6362-422e-9ff1-6f93523ffc01-dshm\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:42.727928 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.727896    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f439b863-dnl22"]
Sep 06 19:43:42.728365 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.728343    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afa9ed92-6362-422e-9ff1-6f93523ffc01" containerName="main"
Sep 06 19:43:42.728365 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.728363    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="afa9ed92-6362-422e-9ff1-6f93523ffc01" containerName="main"
Sep 06 19:43:42.728528 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.728401    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="654441ed-52e4-4ced-87ee-f9e84f662271" containerName="revoke-keys"
Sep 06 19:43:42.728528 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.728411    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="654441ed-52e4-4ced-87ee-f9e84f662271" containerName="revoke-keys"
Sep 06 19:43:42.728528 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.728490    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="654441ed-52e4-4ced-87ee-f9e84f662271" containerName="revoke-keys"
Sep 06 19:43:42.728528 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.728506    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="afa9ed92-6362-422e-9ff1-6f93523ffc01" containerName="main"
Sep 06 19:43:42.731742 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.731712    2571 generic.go:356] "Generic (PLEG): container finished" podID="afa9ed92-6362-422e-9ff1-6f93523ffc01" containerID="84ade560a206976191fdb7cdc0accaa829ca39440df8831dff984cb40fd929ca" exitCode=0
Sep 06 19:43:42.731844 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.731777    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"
Sep 06 19:43:42.731923 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.731894    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc" event={"ID":"afa9ed92-6362-422e-9ff1-6f93523ffc01","Type":"ContainerDied","Data":"84ade560a206976191fdb7cdc0accaa829ca39440df8831dff984cb40fd929ca"}
Sep 06 19:43:42.732013 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.731937    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc" event={"ID":"afa9ed92-6362-422e-9ff1-6f93523ffc01","Type":"ContainerDied","Data":"381ab15efdc9fa5ad38da424fcfe291110e4e40c339e264c3f202ca9d16f6bfa"}
Sep 06 19:43:42.732013 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.731992    2571 scope.go:117] "RemoveContainer" containerID="84ade560a206976191fdb7cdc0accaa829ca39440df8831dff984cb40fd929ca"
Sep 06 19:43:42.732131 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.732062    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f439b863-dnl22"
Sep 06 19:43:42.734472 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.734451    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h45ww\""
Sep 06 19:43:42.734592 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.734514    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 19:43:42.738937 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.738909    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f439b863-dnl22"]
Sep 06 19:43:42.744155 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.744131    2571 scope.go:117] "RemoveContainer" containerID="84ade560a206976191fdb7cdc0accaa829ca39440df8831dff984cb40fd929ca"
Sep 06 19:43:42.744578 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:43:42.744523    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ade560a206976191fdb7cdc0accaa829ca39440df8831dff984cb40fd929ca\": container with ID starting with 84ade560a206976191fdb7cdc0accaa829ca39440df8831dff984cb40fd929ca not found: ID does not exist" containerID="84ade560a206976191fdb7cdc0accaa829ca39440df8831dff984cb40fd929ca"
Sep 06 19:43:42.744578 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.744560    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ade560a206976191fdb7cdc0accaa829ca39440df8831dff984cb40fd929ca"} err="failed to get container status \"84ade560a206976191fdb7cdc0accaa829ca39440df8831dff984cb40fd929ca\": rpc error: code = NotFound desc = could not find container \"84ade560a206976191fdb7cdc0accaa829ca39440df8831dff984cb40fd929ca\": container with ID starting with 84ade560a206976191fdb7cdc0accaa829ca39440df8831dff984cb40fd929ca not found: ID does not exist"
Sep 06 19:43:42.760555 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.760523    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"]
Sep 06 19:43:42.765900 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.765872    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-6e7ee48c-kserve-67ffd4dd4b-wjkwc"]
Sep 06 19:43:42.827127 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.827090    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884dc240-1ad1-4dd1-a0b1-afb2afc67e73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f439b863-dnl22\" (UID: \"884dc240-1ad1-4dd1-a0b1-afb2afc67e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f439b863-dnl22"
Sep 06 19:43:42.918554 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.918520    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afa9ed92-6362-422e-9ff1-6f93523ffc01" path="/var/lib/kubelet/pods/afa9ed92-6362-422e-9ff1-6f93523ffc01/volumes"
Sep 06 19:43:42.927852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.927828    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884dc240-1ad1-4dd1-a0b1-afb2afc67e73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f439b863-dnl22\" (UID: \"884dc240-1ad1-4dd1-a0b1-afb2afc67e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f439b863-dnl22"
Sep 06 19:43:42.928474 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:42.928455    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884dc240-1ad1-4dd1-a0b1-afb2afc67e73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f439b863-dnl22\" (UID: \"884dc240-1ad1-4dd1-a0b1-afb2afc67e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f439b863-dnl22"
Sep 06 19:43:43.047900 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:43.047806    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f439b863-dnl22"
Sep 06 19:43:43.318661 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:43.318635    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f439b863-dnl22"]
Sep 06 19:43:43.320524 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:43:43.320496    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884dc240_1ad1_4dd1_a0b1_afb2afc67e73.slice/crio-bc16ba328e991f811b3bfbbb4768878f0c678f4405b74165e1342b0996279c5c WatchSource:0}: Error finding container bc16ba328e991f811b3bfbbb4768878f0c678f4405b74165e1342b0996279c5c: Status 404 returned error can't find the container with id bc16ba328e991f811b3bfbbb4768878f0c678f4405b74165e1342b0996279c5c
Sep 06 19:43:43.738884 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:43.738845    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f439b863-dnl22" event={"ID":"884dc240-1ad1-4dd1-a0b1-afb2afc67e73","Type":"ContainerStarted","Data":"bc16ba328e991f811b3bfbbb4768878f0c678f4405b74165e1342b0996279c5c"}
Sep 06 19:43:43.797291 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:43.797249    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj"]
Sep 06 19:43:43.799532 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:43.799507    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0ebbc9-cszcj"]
Sep 06 19:43:44.254245 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.254206    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb" podUID="807122d4-6fc3-4afb-b636-092cec85d529" containerName="main" containerID="cri-o://98a88bf59456e4fb10d355771414697d9450338b3228deb54933a1076638f2f8" gracePeriod=45
Sep 06 19:43:44.502696 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.502673    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:43:44.644844 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.644806    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-home\") pod \"807122d4-6fc3-4afb-b636-092cec85d529\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") "
Sep 06 19:43:44.645068 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.644857    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-tmp-dir\") pod \"807122d4-6fc3-4afb-b636-092cec85d529\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") "
Sep 06 19:43:44.645068 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.644891    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/807122d4-6fc3-4afb-b636-092cec85d529-tls-certs\") pod \"807122d4-6fc3-4afb-b636-092cec85d529\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") "
Sep 06 19:43:44.645068 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.644940    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-model-cache\") pod \"807122d4-6fc3-4afb-b636-092cec85d529\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") "
Sep 06 19:43:44.645068 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.645049    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-dshm\") pod \"807122d4-6fc3-4afb-b636-092cec85d529\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") "
Sep 06 19:43:44.645282 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.645081    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsggv\" (UniqueName: \"kubernetes.io/projected/807122d4-6fc3-4afb-b636-092cec85d529-kube-api-access-rsggv\") pod \"807122d4-6fc3-4afb-b636-092cec85d529\" (UID: \"807122d4-6fc3-4afb-b636-092cec85d529\") "
Sep 06 19:43:44.645282 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.645139    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-home" (OuterVolumeSpecName: "home") pod "807122d4-6fc3-4afb-b636-092cec85d529" (UID: "807122d4-6fc3-4afb-b636-092cec85d529"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:44.645282 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.645168    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "807122d4-6fc3-4afb-b636-092cec85d529" (UID: "807122d4-6fc3-4afb-b636-092cec85d529"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:44.645282 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.645212    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-model-cache" (OuterVolumeSpecName: "model-cache") pod "807122d4-6fc3-4afb-b636-092cec85d529" (UID: "807122d4-6fc3-4afb-b636-092cec85d529"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:44.645422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.645406    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-home\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:44.645422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.645419    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-tmp-dir\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:44.645530 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.645428    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-model-cache\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:44.647013 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.646987    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/807122d4-6fc3-4afb-b636-092cec85d529-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "807122d4-6fc3-4afb-b636-092cec85d529" (UID: "807122d4-6fc3-4afb-b636-092cec85d529"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:43:44.647236 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.647200    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807122d4-6fc3-4afb-b636-092cec85d529-kube-api-access-rsggv" (OuterVolumeSpecName: "kube-api-access-rsggv") pod "807122d4-6fc3-4afb-b636-092cec85d529" (UID: "807122d4-6fc3-4afb-b636-092cec85d529"). InnerVolumeSpecName "kube-api-access-rsggv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:43:44.647349 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.647245    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-dshm" (OuterVolumeSpecName: "dshm") pod "807122d4-6fc3-4afb-b636-092cec85d529" (UID: "807122d4-6fc3-4afb-b636-092cec85d529"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:44.746105 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.746066    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/807122d4-6fc3-4afb-b636-092cec85d529-tls-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:44.746105 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.746100    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/807122d4-6fc3-4afb-b636-092cec85d529-dshm\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:44.746293 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.746114    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsggv\" (UniqueName: \"kubernetes.io/projected/807122d4-6fc3-4afb-b636-092cec85d529-kube-api-access-rsggv\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:44.747187 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.747161    2571 generic.go:356] "Generic (PLEG): container finished" podID="884dc240-1ad1-4dd1-a0b1-afb2afc67e73" containerID="0cafe4b673068ea14a922ff399f35ab41b36edcef3cfe506128d2340fcec3384" exitCode=0
Sep 06 19:43:44.747324 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.747245    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f439b863-dnl22" event={"ID":"884dc240-1ad1-4dd1-a0b1-afb2afc67e73","Type":"ContainerDied","Data":"0cafe4b673068ea14a922ff399f35ab41b36edcef3cfe506128d2340fcec3384"}
Sep 06 19:43:44.748561 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.748542    2571 generic.go:356] "Generic (PLEG): container finished" podID="807122d4-6fc3-4afb-b636-092cec85d529" containerID="98a88bf59456e4fb10d355771414697d9450338b3228deb54933a1076638f2f8" exitCode=0
Sep 06 19:43:44.748640 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.748576    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"
Sep 06 19:43:44.748640 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.748611    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb" event={"ID":"807122d4-6fc3-4afb-b636-092cec85d529","Type":"ContainerDied","Data":"98a88bf59456e4fb10d355771414697d9450338b3228deb54933a1076638f2f8"}
Sep 06 19:43:44.748640 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.748638    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb" event={"ID":"807122d4-6fc3-4afb-b636-092cec85d529","Type":"ContainerDied","Data":"74d569a8be5c76b54f33bed2a14ea527cbe6fbb8e16d91ec0a311dcaef00dc23"}
Sep 06 19:43:44.748749 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.748654    2571 scope.go:117] "RemoveContainer" containerID="98a88bf59456e4fb10d355771414697d9450338b3228deb54933a1076638f2f8"
Sep 06 19:43:44.757746 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.757729    2571 scope.go:117] "RemoveContainer" containerID="98a88bf59456e4fb10d355771414697d9450338b3228deb54933a1076638f2f8"
Sep 06 19:43:44.758051 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:43:44.758031    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a88bf59456e4fb10d355771414697d9450338b3228deb54933a1076638f2f8\": container with ID starting with 98a88bf59456e4fb10d355771414697d9450338b3228deb54933a1076638f2f8 not found: ID does not exist" containerID="98a88bf59456e4fb10d355771414697d9450338b3228deb54933a1076638f2f8"
Sep 06 19:43:44.758120 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.758058    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a88bf59456e4fb10d355771414697d9450338b3228deb54933a1076638f2f8"} err="failed to get container status \"98a88bf59456e4fb10d355771414697d9450338b3228deb54933a1076638f2f8\": rpc error: code = NotFound desc = could not find container \"98a88bf59456e4fb10d355771414697d9450338b3228deb54933a1076638f2f8\": container with ID starting with 98a88bf59456e4fb10d355771414697d9450338b3228deb54933a1076638f2f8 not found: ID does not exist"
Sep 06 19:43:44.778394 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.778357    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"]
Sep 06 19:43:44.779372 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.779340    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-6e7ee48c-kserve-798789fb-6t6pb"]
Sep 06 19:43:44.918461 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.918380    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="654441ed-52e4-4ced-87ee-f9e84f662271" path="/var/lib/kubelet/pods/654441ed-52e4-4ced-87ee-f9e84f662271/volumes"
Sep 06 19:43:44.918810 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:44.918795    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="807122d4-6fc3-4afb-b636-092cec85d529" path="/var/lib/kubelet/pods/807122d4-6fc3-4afb-b636-092cec85d529/volumes"
Sep 06 19:43:45.882038 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:45.882010    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f439b863-dnl22"
Sep 06 19:43:45.955053 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:45.955020    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884dc240-1ad1-4dd1-a0b1-afb2afc67e73-maas-api-service-ca\") pod \"884dc240-1ad1-4dd1-a0b1-afb2afc67e73\" (UID: \"884dc240-1ad1-4dd1-a0b1-afb2afc67e73\") "
Sep 06 19:43:45.955380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:45.955356    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884dc240-1ad1-4dd1-a0b1-afb2afc67e73-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "884dc240-1ad1-4dd1-a0b1-afb2afc67e73" (UID: "884dc240-1ad1-4dd1-a0b1-afb2afc67e73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:43:46.056244 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:46.056156    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884dc240-1ad1-4dd1-a0b1-afb2afc67e73-maas-api-service-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:46.762104 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:46.762065    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f439b863-dnl22" event={"ID":"884dc240-1ad1-4dd1-a0b1-afb2afc67e73","Type":"ContainerDied","Data":"bc16ba328e991f811b3bfbbb4768878f0c678f4405b74165e1342b0996279c5c"}
Sep 06 19:43:46.762104 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:46.762091    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f439b863-dnl22"
Sep 06 19:43:46.762387 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:46.762127    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc16ba328e991f811b3bfbbb4768878f0c678f4405b74165e1342b0996279c5c"
Sep 06 19:43:47.279389 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.279345    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t" podUID="026ef435-1597-441a-a300-fcc8873f8c5d" containerName="main" containerID="cri-o://c4e219b014fcf2c095e12112b8f9b7bc60c9c04efa1acc83d00776152874c415" gracePeriod=45
Sep 06 19:43:47.523610 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.523581    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:47.568182 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.568153    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026ef435-1597-441a-a300-fcc8873f8c5d-tls-certs\") pod \"026ef435-1597-441a-a300-fcc8873f8c5d\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") "
Sep 06 19:43:47.568380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.568202    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-dshm\") pod \"026ef435-1597-441a-a300-fcc8873f8c5d\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") "
Sep 06 19:43:47.568380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.568256    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-model-cache\") pod \"026ef435-1597-441a-a300-fcc8873f8c5d\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") "
Sep 06 19:43:47.568380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.568306    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-home\") pod \"026ef435-1597-441a-a300-fcc8873f8c5d\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") "
Sep 06 19:43:47.568380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.568355    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqf57\" (UniqueName: \"kubernetes.io/projected/026ef435-1597-441a-a300-fcc8873f8c5d-kube-api-access-wqf57\") pod \"026ef435-1597-441a-a300-fcc8873f8c5d\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") "
Sep 06 19:43:47.568592 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.568383    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-tmp-dir\") pod \"026ef435-1597-441a-a300-fcc8873f8c5d\" (UID: \"026ef435-1597-441a-a300-fcc8873f8c5d\") "
Sep 06 19:43:47.568592 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.568544    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-model-cache" (OuterVolumeSpecName: "model-cache") pod "026ef435-1597-441a-a300-fcc8873f8c5d" (UID: "026ef435-1597-441a-a300-fcc8873f8c5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:47.568705 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.568592    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-home" (OuterVolumeSpecName: "home") pod "026ef435-1597-441a-a300-fcc8873f8c5d" (UID: "026ef435-1597-441a-a300-fcc8873f8c5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:47.568705 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.568639    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "026ef435-1597-441a-a300-fcc8873f8c5d" (UID: "026ef435-1597-441a-a300-fcc8873f8c5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:47.568822 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.568806    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-model-cache\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:47.568882 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.568823    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-home\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:47.568882 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.568832    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-tmp-dir\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:47.570607 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.570574    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026ef435-1597-441a-a300-fcc8873f8c5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "026ef435-1597-441a-a300-fcc8873f8c5d" (UID: "026ef435-1597-441a-a300-fcc8873f8c5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:43:47.570711 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.570634    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026ef435-1597-441a-a300-fcc8873f8c5d-kube-api-access-wqf57" (OuterVolumeSpecName: "kube-api-access-wqf57") pod "026ef435-1597-441a-a300-fcc8873f8c5d" (UID: "026ef435-1597-441a-a300-fcc8873f8c5d"). InnerVolumeSpecName "kube-api-access-wqf57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:43:47.570711 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.570647    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-dshm" (OuterVolumeSpecName: "dshm") pod "026ef435-1597-441a-a300-fcc8873f8c5d" (UID: "026ef435-1597-441a-a300-fcc8873f8c5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:47.670190 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.670151    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026ef435-1597-441a-a300-fcc8873f8c5d-tls-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:47.670190 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.670182    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026ef435-1597-441a-a300-fcc8873f8c5d-dshm\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:47.670190 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.670192    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqf57\" (UniqueName: \"kubernetes.io/projected/026ef435-1597-441a-a300-fcc8873f8c5d-kube-api-access-wqf57\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:47.769144 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.769114    2571 generic.go:356] "Generic (PLEG): container finished" podID="026ef435-1597-441a-a300-fcc8873f8c5d" containerID="c4e219b014fcf2c095e12112b8f9b7bc60c9c04efa1acc83d00776152874c415" exitCode=0
Sep 06 19:43:47.769293 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.769159    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"
Sep 06 19:43:47.769293 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.769182    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t" event={"ID":"026ef435-1597-441a-a300-fcc8873f8c5d","Type":"ContainerDied","Data":"c4e219b014fcf2c095e12112b8f9b7bc60c9c04efa1acc83d00776152874c415"}
Sep 06 19:43:47.769293 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.769214    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t" event={"ID":"026ef435-1597-441a-a300-fcc8873f8c5d","Type":"ContainerDied","Data":"1f15bd7f7b3988ec7047ba4650c75baaf4fcd4699aaf2d9b997d81d279739a13"}
Sep 06 19:43:47.769293 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.769230    2571 scope.go:117] "RemoveContainer" containerID="c4e219b014fcf2c095e12112b8f9b7bc60c9c04efa1acc83d00776152874c415"
Sep 06 19:43:47.778912 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.778892    2571 scope.go:117] "RemoveContainer" containerID="c4e219b014fcf2c095e12112b8f9b7bc60c9c04efa1acc83d00776152874c415"
Sep 06 19:43:47.779213 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:43:47.779193    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e219b014fcf2c095e12112b8f9b7bc60c9c04efa1acc83d00776152874c415\": container with ID starting with c4e219b014fcf2c095e12112b8f9b7bc60c9c04efa1acc83d00776152874c415 not found: ID does not exist" containerID="c4e219b014fcf2c095e12112b8f9b7bc60c9c04efa1acc83d00776152874c415"
Sep 06 19:43:47.779267 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.779221    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e219b014fcf2c095e12112b8f9b7bc60c9c04efa1acc83d00776152874c415"} err="failed to get container status \"c4e219b014fcf2c095e12112b8f9b7bc60c9c04efa1acc83d00776152874c415\": rpc error: code = NotFound desc = could not find container \"c4e219b014fcf2c095e12112b8f9b7bc60c9c04efa1acc83d00776152874c415\": container with ID starting with c4e219b014fcf2c095e12112b8f9b7bc60c9c04efa1acc83d00776152874c415 not found: ID does not exist"
Sep 06 19:43:47.789582 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.789554    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"]
Sep 06 19:43:47.792719 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:47.792694    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-0ebbc9/autores-model-0ebbc9-kserve-665c6cbbdc-ls44t"]
Sep 06 19:43:48.918862 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:48.918828    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="026ef435-1597-441a-a300-fcc8873f8c5d" path="/var/lib/kubelet/pods/026ef435-1597-441a-a300-fcc8873f8c5d/volumes"
Sep 06 19:43:49.805735 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:49.805699    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f439b863-dnl22"]
Sep 06 19:43:49.809479 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:49.809452    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f439b863-dnl22"]
Sep 06 19:43:50.642346 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:50.642305    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"]
Sep 06 19:43:50.642825 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:50.642543    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd" podUID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerName="istio-proxy" containerID="cri-o://105dfc3ffadfb893f49e6d9bb61f7dd046444e038b17b7a3dd91b0939d8a22d8" gracePeriod=30
Sep 06 19:43:50.918698 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:50.918621    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="884dc240-1ad1-4dd1-a0b1-afb2afc67e73" path="/var/lib/kubelet/pods/884dc240-1ad1-4dd1-a0b1-afb2afc67e73/volumes"
Sep 06 19:43:51.338878 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.338846    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"]
Sep 06 19:43:51.339477 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.339456    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="884dc240-1ad1-4dd1-a0b1-afb2afc67e73" containerName="revoke-keys"
Sep 06 19:43:51.339525 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.339484    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="884dc240-1ad1-4dd1-a0b1-afb2afc67e73" containerName="revoke-keys"
Sep 06 19:43:51.339525 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.339505    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="807122d4-6fc3-4afb-b636-092cec85d529" containerName="main"
Sep 06 19:43:51.339525 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.339514    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="807122d4-6fc3-4afb-b636-092cec85d529" containerName="main"
Sep 06 19:43:51.339624 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.339528    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="026ef435-1597-441a-a300-fcc8873f8c5d" containerName="main"
Sep 06 19:43:51.339624 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.339537    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="026ef435-1597-441a-a300-fcc8873f8c5d" containerName="main"
Sep 06 19:43:51.339688 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.339624    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="807122d4-6fc3-4afb-b636-092cec85d529" containerName="main"
Sep 06 19:43:51.339688 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.339638    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="026ef435-1597-441a-a300-fcc8873f8c5d" containerName="main"
Sep 06 19:43:51.339688 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.339647    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="884dc240-1ad1-4dd1-a0b1-afb2afc67e73" containerName="revoke-keys"
Sep 06 19:43:51.344700 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.344679    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.346888 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.346867    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-838def-openshift-default-dockercfg-8srwz\""
Sep 06 19:43:51.357323 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.357299    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"]
Sep 06 19:43:51.407019 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.406979    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-workload-socket\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.407186 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.407044    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-workload-certs\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.407186 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.407064    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istiod-ca-cert\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.407186 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.407084    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-podinfo\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.407186 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.407111    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-token\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.407186 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.407147    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-envoy\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.407186 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.407185    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-data\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.407424 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.407261    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-credential-socket\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.407424 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.407297    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8cn2\" (UniqueName: \"kubernetes.io/projected/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-kube-api-access-d8cn2\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.508020 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.507985    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-token\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.508220 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.508040    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-envoy\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.508220 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.508086    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-data\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.508220 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.508111    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-credential-socket\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.508220 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.508128    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8cn2\" (UniqueName: \"kubernetes.io/projected/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-kube-api-access-d8cn2\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.508220 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.508147    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-workload-socket\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.508220 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.508211    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-workload-certs\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.508539 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.508242    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istiod-ca-cert\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.508539 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.508274    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-podinfo\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.508780 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.508745    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-credential-socket\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.508906 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.508781    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-workload-certs\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.508906 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.508807    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-workload-socket\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.509104 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.509079    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istiod-ca-cert\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.509172 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.509125    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-data\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.510660 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.510622    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-envoy\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.510795 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.510776    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-podinfo\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.515646 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.515617    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-token\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.515752 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.515724    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8cn2\" (UniqueName: \"kubernetes.io/projected/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-kube-api-access-d8cn2\") pod \"e2e-explicit-838def-openshift-default-54444657cb-flr2g\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") " pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.661404 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.661323    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:51.936379 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.936349    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"]
Sep 06 19:43:51.938482 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:43:51.938447    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b29af8_0ce2_4d82_881e_ae3de5817fa3.slice/crio-48c723c9ba32bc401d482fc0f3d6ee442a4cb7066bfd0d9a94ef2ce7a0e82835 WatchSource:0}: Error finding container 48c723c9ba32bc401d482fc0f3d6ee442a4cb7066bfd0d9a94ef2ce7a0e82835: Status 404 returned error can't find the container with id 48c723c9ba32bc401d482fc0f3d6ee442a4cb7066bfd0d9a94ef2ce7a0e82835
Sep 06 19:43:51.940718 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.940680    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:43:51.940792 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.940743    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:43:51.940792 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:51.940772    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:43:52.800287 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:52.800244    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g" event={"ID":"e3b29af8-0ce2-4d82-881e-ae3de5817fa3","Type":"ContainerStarted","Data":"25c10507f2581dc3ee96342e2d6afb103370ebb310cbb4b79a112d2811696109"}
Sep 06 19:43:52.800675 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:52.800298    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g" event={"ID":"e3b29af8-0ce2-4d82-881e-ae3de5817fa3","Type":"ContainerStarted","Data":"48c723c9ba32bc401d482fc0f3d6ee442a4cb7066bfd0d9a94ef2ce7a0e82835"}
Sep 06 19:43:52.820257 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:52.820203    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g" podStartSLOduration=1.820182789 podStartE2EDuration="1.820182789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:43:52.819950133 +0000 UTC m=+996.390248480" watchObservedRunningTime="2026-09-06 19:43:52.820182789 +0000 UTC m=+996.390481136"
Sep 06 19:43:53.661684 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:53.661647    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:53.666688 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:53.666661    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:53.805216 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:53.805186    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:53.806133 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:53.806117    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:43:55.821297 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.821265    2571 generic.go:356] "Generic (PLEG): container finished" podID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerID="105dfc3ffadfb893f49e6d9bb61f7dd046444e038b17b7a3dd91b0939d8a22d8" exitCode=0
Sep 06 19:43:55.822089 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.821341    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd" event={"ID":"025a5d67-7c5a-4b97-9d22-c3112e7a1c94","Type":"ContainerDied","Data":"105dfc3ffadfb893f49e6d9bb61f7dd046444e038b17b7a3dd91b0939d8a22d8"}
Sep 06 19:43:55.822089 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.821443    2571 scope.go:117] "RemoveContainer" containerID="5f3765524bb43e3b1ad73a48208811c1f531f3a826057b9edcf33bab1f73ac5b"
Sep 06 19:43:55.910923 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.910848    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:43:55.945459 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.945426    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-podinfo\") pod \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") "
Sep 06 19:43:55.945641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.945504    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-data\") pod \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") "
Sep 06 19:43:55.945641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.945529    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-workload-socket\") pod \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") "
Sep 06 19:43:55.945641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.945545    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-envoy\") pod \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") "
Sep 06 19:43:55.945641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.945570    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-token\") pod \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") "
Sep 06 19:43:55.945641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.945629    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-credential-socket\") pod \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") "
Sep 06 19:43:55.945925 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.945661    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54trq\" (UniqueName: \"kubernetes.io/projected/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-kube-api-access-54trq\") pod \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") "
Sep 06 19:43:55.945925 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.945696    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-workload-certs\") pod \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") "
Sep 06 19:43:55.945925 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.945734    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istiod-ca-cert\") pod \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\" (UID: \"025a5d67-7c5a-4b97-9d22-c3112e7a1c94\") "
Sep 06 19:43:55.945925 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.945817    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "025a5d67-7c5a-4b97-9d22-c3112e7a1c94" (UID: "025a5d67-7c5a-4b97-9d22-c3112e7a1c94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:55.945925 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.945842    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-data" (OuterVolumeSpecName: "istio-data") pod "025a5d67-7c5a-4b97-9d22-c3112e7a1c94" (UID: "025a5d67-7c5a-4b97-9d22-c3112e7a1c94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:55.945925 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.945906    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "025a5d67-7c5a-4b97-9d22-c3112e7a1c94" (UID: "025a5d67-7c5a-4b97-9d22-c3112e7a1c94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:55.946277 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.946039    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "025a5d67-7c5a-4b97-9d22-c3112e7a1c94" (UID: "025a5d67-7c5a-4b97-9d22-c3112e7a1c94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:55.946277 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.946141    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "025a5d67-7c5a-4b97-9d22-c3112e7a1c94" (UID: "025a5d67-7c5a-4b97-9d22-c3112e7a1c94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:43:55.946277 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.946157    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-data\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:55.946277 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.946176    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-workload-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:55.946277 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.946186    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-credential-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:55.946277 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.946196    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-workload-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:55.947806 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.947780    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "025a5d67-7c5a-4b97-9d22-c3112e7a1c94" (UID: "025a5d67-7c5a-4b97-9d22-c3112e7a1c94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 19:43:55.948123 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.948105    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-token" (OuterVolumeSpecName: "istio-token") pod "025a5d67-7c5a-4b97-9d22-c3112e7a1c94" (UID: "025a5d67-7c5a-4b97-9d22-c3112e7a1c94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:43:55.948241 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.948219    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "025a5d67-7c5a-4b97-9d22-c3112e7a1c94" (UID: "025a5d67-7c5a-4b97-9d22-c3112e7a1c94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:43:55.948307 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:55.948236    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-kube-api-access-54trq" (OuterVolumeSpecName: "kube-api-access-54trq") pod "025a5d67-7c5a-4b97-9d22-c3112e7a1c94" (UID: "025a5d67-7c5a-4b97-9d22-c3112e7a1c94"). InnerVolumeSpecName "kube-api-access-54trq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:43:56.046675 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:56.046639    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-envoy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:56.046675 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:56.046671    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-token\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:56.046675 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:56.046680    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54trq\" (UniqueName: \"kubernetes.io/projected/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-kube-api-access-54trq\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:56.046912 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:56.046689    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istiod-ca-cert\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:56.046912 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:56.046698    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025a5d67-7c5a-4b97-9d22-c3112e7a1c94-istio-podinfo\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:43:56.828157 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:56.828129    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"
Sep 06 19:43:56.828572 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:56.828175    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd" event={"ID":"025a5d67-7c5a-4b97-9d22-c3112e7a1c94","Type":"ContainerDied","Data":"3796e1aa10d991c3f78a1d3e46424452c0b8d659a477eb0e901072a670a80c34"}
Sep 06 19:43:56.828572 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:56.828233    2571 scope.go:117] "RemoveContainer" containerID="105dfc3ffadfb893f49e6d9bb61f7dd046444e038b17b7a3dd91b0939d8a22d8"
Sep 06 19:43:56.849643 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:56.849608    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"]
Sep 06 19:43:56.854437 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:56.854411    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-0ebbc9-openshift-default-765dc488cb-tpvbd"]
Sep 06 19:43:56.920227 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:56.920194    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" path="/var/lib/kubelet/pods/025a5d67-7c5a-4b97-9d22-c3112e7a1c94/volumes"
Sep 06 19:43:58.757040 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:43:58.757004    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:44:04.327249 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.327211    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"]
Sep 06 19:44:04.327660 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.327600    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerName="istio-proxy"
Sep 06 19:44:04.327660 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.327611    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerName="istio-proxy"
Sep 06 19:44:04.327660 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.327620    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerName="istio-proxy"
Sep 06 19:44:04.327660 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.327625    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerName="istio-proxy"
Sep 06 19:44:04.327789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.327691    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerName="istio-proxy"
Sep 06 19:44:04.327789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.327699    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="025a5d67-7c5a-4b97-9d22-c3112e7a1c94" containerName="istio-proxy"
Sep 06 19:44:04.332357 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.332337    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.334720 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.334695    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d5bb2d\"/\"kube-root-ca.crt\""
Sep 06 19:44:04.335989 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.335950    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d5bb2d\"/\"default-dockercfg-tl2m6\""
Sep 06 19:44:04.336126 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.335971    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d5bb2d\"/\"openshift-service-ca.crt\""
Sep 06 19:44:04.336126 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.336090    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d5bb2d\"/\"test-model-d5bb2d-kserve-self-signed-certs\""
Sep 06 19:44:04.338749 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.338724    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"]
Sep 06 19:44:04.426875 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.426843    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-dshm\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.427050 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.426885    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-model-cache\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.427050 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.426915    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-tmp-dir\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.427050 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.427005    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae277b9f-bca2-4a4b-9da5-7fba9160acee-tls-certs\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.427206 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.427050    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-home\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.427206 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.427082    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b4c9\" (UniqueName: \"kubernetes.io/projected/ae277b9f-bca2-4a4b-9da5-7fba9160acee-kube-api-access-4b4c9\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.527549 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.527515    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-home\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.527738 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.527558    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b4c9\" (UniqueName: \"kubernetes.io/projected/ae277b9f-bca2-4a4b-9da5-7fba9160acee-kube-api-access-4b4c9\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.527738 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.527595    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-dshm\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.527738 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.527624    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-model-cache\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.527738 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.527652    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-tmp-dir\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.527738 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.527680    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae277b9f-bca2-4a4b-9da5-7fba9160acee-tls-certs\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.528047 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.528012    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-home\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.528098 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.528051    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-model-cache\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.528155 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.528136    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-tmp-dir\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.529821 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.529803    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-dshm\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.530107 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.530089    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae277b9f-bca2-4a4b-9da5-7fba9160acee-tls-certs\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.534636 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.534614    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b4c9\" (UniqueName: \"kubernetes.io/projected/ae277b9f-bca2-4a4b-9da5-7fba9160acee-kube-api-access-4b4c9\") pod \"test-model-d5bb2d-kserve-bf5b7b75-dmxmb\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") " pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.642980 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.642885    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:04.911591 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:04.911561    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"]
Sep 06 19:44:04.913715 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:44:04.913683    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae277b9f_bca2_4a4b_9da5_7fba9160acee.slice/crio-d4446f4bdcc7593f2260ecc555d0c4e7ebc7cea486c434bad03202756ce62068 WatchSource:0}: Error finding container d4446f4bdcc7593f2260ecc555d0c4e7ebc7cea486c434bad03202756ce62068: Status 404 returned error can't find the container with id d4446f4bdcc7593f2260ecc555d0c4e7ebc7cea486c434bad03202756ce62068
Sep 06 19:44:05.884172 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:05.884132    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb" event={"ID":"ae277b9f-bca2-4a4b-9da5-7fba9160acee","Type":"ContainerStarted","Data":"518b7f01ee5c1dc313e8f7364d49d2c69f44f7b81c43ba0b7fa568382d5e0dd0"}
Sep 06 19:44:05.884681 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:05.884186    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb" event={"ID":"ae277b9f-bca2-4a4b-9da5-7fba9160acee","Type":"ContainerStarted","Data":"d4446f4bdcc7593f2260ecc555d0c4e7ebc7cea486c434bad03202756ce62068"}
Sep 06 19:44:05.900360 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:05.900289    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb" podStartSLOduration=1.900274735 podStartE2EDuration="1.900274735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:44:05.899138992 +0000 UTC m=+1009.469437337" watchObservedRunningTime="2026-09-06 19:44:05.900274735 +0000 UTC m=+1009.470573081"
Sep 06 19:44:14.643403 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:14.643355    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:14.644040 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:14.643423    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:14.668792 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:14.668762    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:14.942205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:14.942114    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:44:16.438007 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:16.437940    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:44:18.955066 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:18.955031    2571 generic.go:356] "Generic (PLEG): container finished" podID="e3b29af8-0ce2-4d82-881e-ae3de5817fa3" containerID="25c10507f2581dc3ee96342e2d6afb103370ebb310cbb4b79a112d2811696109" exitCode=0
Sep 06 19:44:18.955489 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:18.955084    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g" event={"ID":"e3b29af8-0ce2-4d82-881e-ae3de5817fa3","Type":"ContainerDied","Data":"25c10507f2581dc3ee96342e2d6afb103370ebb310cbb4b79a112d2811696109"}
Sep 06 19:44:18.955489 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:18.955475    2571 scope.go:117] "RemoveContainer" containerID="25c10507f2581dc3ee96342e2d6afb103370ebb310cbb4b79a112d2811696109"
Sep 06 19:44:18.956656 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:18.956621    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:44:18.956715 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:18.956701    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:44:18.956751 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:18.956734    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:44:19.963522 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:19.963487    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g" event={"ID":"e3b29af8-0ce2-4d82-881e-ae3de5817fa3","Type":"ContainerStarted","Data":"253acdee6e6791e284cc0c97c5472aab4d3706d99736dc15a40cc1f53f1be74d"}
Sep 06 19:44:20.662075 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:20.662032    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:44:20.663248 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:20.663220    2571 patch_prober.go:28] interesting pod/e2e-explicit-838def-openshift-default-54444657cb-flr2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 06 19:44:20.663381 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:20.663270    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g" podUID="e3b29af8-0ce2-4d82-881e-ae3de5817fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 06 19:44:21.662708 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:21.662664    2571 patch_prober.go:28] interesting pod/e2e-explicit-838def-openshift-default-54444657cb-flr2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 06 19:44:21.663159 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:21.662739    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g" podUID="e3b29af8-0ce2-4d82-881e-ae3de5817fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 06 19:44:22.661833 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:22.661794    2571 patch_prober.go:28] interesting pod/e2e-explicit-838def-openshift-default-54444657cb-flr2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 06 19:44:22.662080 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:22.661852    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g" podUID="e3b29af8-0ce2-4d82-881e-ae3de5817fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 06 19:44:23.662160 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:23.662117    2571 patch_prober.go:28] interesting pod/e2e-explicit-838def-openshift-default-54444657cb-flr2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 06 19:44:23.662545 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:23.662188    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g" podUID="e3b29af8-0ce2-4d82-881e-ae3de5817fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 06 19:44:23.805476 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:23.805432    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:44:24.662512 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:24.662056    2571 patch_prober.go:28] interesting pod/e2e-explicit-838def-openshift-default-54444657cb-flr2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 06 19:44:24.662512 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:24.662123    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g" podUID="e3b29af8-0ce2-4d82-881e-ae3de5817fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 06 19:44:25.665699 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:25.665668    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:44:25.666913 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:25.666890    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:44:32.243681 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:32.243648    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:44:46.641631 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:46.641595    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:44:50.737646 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.737605    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"]
Sep 06 19:44:50.741195 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.741178    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.744597 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.744573    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-838def\"/\"default-dockercfg-sldnp\""
Sep 06 19:44:50.744762 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.744744    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-838def\"/\"kube-root-ca.crt\""
Sep 06 19:44:50.744861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.744777    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-838def\"/\"explicit-model-838def-kserve-self-signed-certs\""
Sep 06 19:44:50.744861 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.744797    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-838def\"/\"openshift-service-ca.crt\""
Sep 06 19:44:50.750951 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.750926    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"]
Sep 06 19:44:50.789478 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.789441    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-home\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.789659 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.789498    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-dshm\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.789659 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.789519    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ab527b-de71-46ff-9368-46a0f74ab759-tls-certs\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.789659 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.789636    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-tmp-dir\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.789797 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.789665    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qs4h\" (UniqueName: \"kubernetes.io/projected/b2ab527b-de71-46ff-9368-46a0f74ab759-kube-api-access-6qs4h\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.789797 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.789691    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-model-cache\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.891008 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.890937    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-dshm\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.891213 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.891040    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ab527b-de71-46ff-9368-46a0f74ab759-tls-certs\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.891213 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.891109    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-tmp-dir\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.891213 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.891137    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qs4h\" (UniqueName: \"kubernetes.io/projected/b2ab527b-de71-46ff-9368-46a0f74ab759-kube-api-access-6qs4h\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.891213 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.891175    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-model-cache\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.891433 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.891257    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-home\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.891526 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.891496    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-tmp-dir\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.891649 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.891563    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-model-cache\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.891649 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.891577    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-home\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.893346 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.893325    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-dshm\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.893504 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.893488    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ab527b-de71-46ff-9368-46a0f74ab759-tls-certs\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:50.898292 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:50.898265    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qs4h\" (UniqueName: \"kubernetes.io/projected/b2ab527b-de71-46ff-9368-46a0f74ab759-kube-api-access-6qs4h\") pod \"explicit-model-838def-kserve-659899bf45-t8qms\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") " pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:51.051761 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:51.051663    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:44:51.322645 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:51.322617    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"]
Sep 06 19:44:51.324542 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:44:51.324511    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2ab527b_de71_46ff_9368_46a0f74ab759.slice/crio-aeb665958bbbed5b5babb6dbd43576432c752230d551ce1185883e07b34ff18e WatchSource:0}: Error finding container aeb665958bbbed5b5babb6dbd43576432c752230d551ce1185883e07b34ff18e: Status 404 returned error can't find the container with id aeb665958bbbed5b5babb6dbd43576432c752230d551ce1185883e07b34ff18e
Sep 06 19:44:52.136291 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:52.136257    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms" event={"ID":"b2ab527b-de71-46ff-9368-46a0f74ab759","Type":"ContainerStarted","Data":"982b279d26e8c0a1a989bdad8ea9ac94fae2a09ce004d59dc84badca9bda3d96"}
Sep 06 19:44:52.136680 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:52.136300    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms" event={"ID":"b2ab527b-de71-46ff-9368-46a0f74ab759","Type":"ContainerStarted","Data":"aeb665958bbbed5b5babb6dbd43576432c752230d551ce1185883e07b34ff18e"}
Sep 06 19:44:52.152803 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:52.152744    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms" podStartSLOduration=2.152730221 podStartE2EDuration="2.152730221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:44:52.151468733 +0000 UTC m=+1055.721767079" watchObservedRunningTime="2026-09-06 19:44:52.152730221 +0000 UTC m=+1055.723028568"
Sep 06 19:44:58.144604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:44:58.144563    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:45:00.137235 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:00.137188    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6433a6-29812065-rkj4x"]
Sep 06 19:45:00.141008 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:00.140984    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29812065-glrj6"]
Sep 06 19:45:00.141123 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:00.141043    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6433a6-29812065-rkj4x"
Sep 06 19:45:00.143360 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:00.143341    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h45ww\""
Sep 06 19:45:00.144126 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:00.144111    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812065-glrj6"
Sep 06 19:45:00.150361 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:00.150336    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6433a6-29812065-rkj4x"]
Sep 06 19:45:00.153124 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:00.153095    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29812065-glrj6"]
Sep 06 19:45:00.154778 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:00.154754    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6433a6-29812065-rkj4x"
Sep 06 19:45:00.163062 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:00.163038    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812065-glrj6"
Sep 06 19:45:00.429699 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:00.429674    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6433a6-29812065-rkj4x"]
Sep 06 19:45:00.432193 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:45:00.432162    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989c1c08_e461_4680_9743_96ecbcdd955e.slice/crio-0d484c2f6090a5884821f5cdf0b29e71788e74f06d2d72fad034a440813d5714 WatchSource:0}: Error finding container 0d484c2f6090a5884821f5cdf0b29e71788e74f06d2d72fad034a440813d5714: Status 404 returned error can't find the container with id 0d484c2f6090a5884821f5cdf0b29e71788e74f06d2d72fad034a440813d5714
Sep 06 19:45:00.439416 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:00.439331    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29812065-glrj6"]
Sep 06 19:45:00.441217 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:45:00.441192    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ad3d00_06f8_4cfd_8fbd_386f026f0175.slice/crio-28d8544d714060cb97f71bd3ba8768d05fd957272d16893f5aefb454288b3f1b WatchSource:0}: Error finding container 28d8544d714060cb97f71bd3ba8768d05fd957272d16893f5aefb454288b3f1b: Status 404 returned error can't find the container with id 28d8544d714060cb97f71bd3ba8768d05fd957272d16893f5aefb454288b3f1b
Sep 06 19:45:01.052519 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:01.052432    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:45:01.052519 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:01.052506    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:45:01.065325 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:01.065298    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:45:01.225243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:01.225206    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812065-glrj6" event={"ID":"27ad3d00-06f8-4cfd-8fbd-386f026f0175","Type":"ContainerStarted","Data":"5d77a411bba564e8c03be7f674c6897506bee79081c828bcc3d2d98dc83ec48e"}
Sep 06 19:45:01.225704 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:01.225262    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812065-glrj6" event={"ID":"27ad3d00-06f8-4cfd-8fbd-386f026f0175","Type":"ContainerStarted","Data":"28d8544d714060cb97f71bd3ba8768d05fd957272d16893f5aefb454288b3f1b"}
Sep 06 19:45:01.226743 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:01.226716    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6433a6-29812065-rkj4x" event={"ID":"989c1c08-e461-4680-9743-96ecbcdd955e","Type":"ContainerStarted","Data":"8223c44907d6fd8e905943bf8dc44a602085195613b04ed23d4ef5de2540bad8"}
Sep 06 19:45:01.226860 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:01.226760    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6433a6-29812065-rkj4x" event={"ID":"989c1c08-e461-4680-9743-96ecbcdd955e","Type":"ContainerStarted","Data":"0d484c2f6090a5884821f5cdf0b29e71788e74f06d2d72fad034a440813d5714"}
Sep 06 19:45:01.238713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:01.238680    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:45:01.238945 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:01.238905    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812065-glrj6" podStartSLOduration=1.23889148 podStartE2EDuration="1.23889148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:45:01.238706529 +0000 UTC m=+1064.809004876" watchObservedRunningTime="2026-09-06 19:45:01.23889148 +0000 UTC m=+1064.809189825"
Sep 06 19:45:01.267380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:01.267316    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6433a6-29812065-rkj4x" podStartSLOduration=1.2672929339999999 podStartE2EDuration="1.267292934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:45:01.265858648 +0000 UTC m=+1064.836156995" watchObservedRunningTime="2026-09-06 19:45:01.267292934 +0000 UTC m=+1064.837591284"
Sep 06 19:45:02.233142 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:02.233105    2571 generic.go:356] "Generic (PLEG): container finished" podID="989c1c08-e461-4680-9743-96ecbcdd955e" containerID="8223c44907d6fd8e905943bf8dc44a602085195613b04ed23d4ef5de2540bad8" exitCode=0
Sep 06 19:45:02.233550 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:02.233189    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6433a6-29812065-rkj4x" event={"ID":"989c1c08-e461-4680-9743-96ecbcdd955e","Type":"ContainerDied","Data":"8223c44907d6fd8e905943bf8dc44a602085195613b04ed23d4ef5de2540bad8"}
Sep 06 19:45:02.234750 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:02.234724    2571 generic.go:356] "Generic (PLEG): container finished" podID="27ad3d00-06f8-4cfd-8fbd-386f026f0175" containerID="5d77a411bba564e8c03be7f674c6897506bee79081c828bcc3d2d98dc83ec48e" exitCode=0
Sep 06 19:45:02.234865 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:02.234777    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812065-glrj6" event={"ID":"27ad3d00-06f8-4cfd-8fbd-386f026f0175","Type":"ContainerDied","Data":"5d77a411bba564e8c03be7f674c6897506bee79081c828bcc3d2d98dc83ec48e"}
Sep 06 19:45:03.026444 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:03.026405    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"]
Sep 06 19:45:03.400499 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:03.400476    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6433a6-29812065-rkj4x"
Sep 06 19:45:03.404147 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:03.404121    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812065-glrj6"
Sep 06 19:45:04.246436 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:04.246407    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812065-glrj6"
Sep 06 19:45:04.246436 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:04.246427    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812065-glrj6" event={"ID":"27ad3d00-06f8-4cfd-8fbd-386f026f0175","Type":"ContainerDied","Data":"28d8544d714060cb97f71bd3ba8768d05fd957272d16893f5aefb454288b3f1b"}
Sep 06 19:45:04.246667 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:04.246476    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28d8544d714060cb97f71bd3ba8768d05fd957272d16893f5aefb454288b3f1b"
Sep 06 19:45:04.247730 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:04.247706    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6433a6-29812065-rkj4x" event={"ID":"989c1c08-e461-4680-9743-96ecbcdd955e","Type":"ContainerDied","Data":"0d484c2f6090a5884821f5cdf0b29e71788e74f06d2d72fad034a440813d5714"}
Sep 06 19:45:04.247730 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:04.247728    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6433a6-29812065-rkj4x"
Sep 06 19:45:04.247884 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:04.247753    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d484c2f6090a5884821f5cdf0b29e71788e74f06d2d72fad034a440813d5714"
Sep 06 19:45:10.739042 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:10.738944    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:45:18.262676 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.262621    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms" podUID="b2ab527b-de71-46ff-9368-46a0f74ab759" containerName="main" containerID="cri-o://982b279d26e8c0a1a989bdad8ea9ac94fae2a09ce004d59dc84badca9bda3d96" gracePeriod=45
Sep 06 19:45:18.519331 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.519266    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:45:18.571711 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.571678    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ab527b-de71-46ff-9368-46a0f74ab759-tls-certs\") pod \"b2ab527b-de71-46ff-9368-46a0f74ab759\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") "
Sep 06 19:45:18.571711 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.571717    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-tmp-dir\") pod \"b2ab527b-de71-46ff-9368-46a0f74ab759\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") "
Sep 06 19:45:18.571996 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.571765    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-home\") pod \"b2ab527b-de71-46ff-9368-46a0f74ab759\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") "
Sep 06 19:45:18.571996 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.571816    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qs4h\" (UniqueName: \"kubernetes.io/projected/b2ab527b-de71-46ff-9368-46a0f74ab759-kube-api-access-6qs4h\") pod \"b2ab527b-de71-46ff-9368-46a0f74ab759\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") "
Sep 06 19:45:18.571996 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.571843    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-model-cache\") pod \"b2ab527b-de71-46ff-9368-46a0f74ab759\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") "
Sep 06 19:45:18.571996 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.571903    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-dshm\") pod \"b2ab527b-de71-46ff-9368-46a0f74ab759\" (UID: \"b2ab527b-de71-46ff-9368-46a0f74ab759\") "
Sep 06 19:45:18.572224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.572062    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-home" (OuterVolumeSpecName: "home") pod "b2ab527b-de71-46ff-9368-46a0f74ab759" (UID: "b2ab527b-de71-46ff-9368-46a0f74ab759"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:45:18.572224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.572092    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2ab527b-de71-46ff-9368-46a0f74ab759" (UID: "b2ab527b-de71-46ff-9368-46a0f74ab759"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:45:18.572328 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.572262    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2ab527b-de71-46ff-9368-46a0f74ab759" (UID: "b2ab527b-de71-46ff-9368-46a0f74ab759"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:45:18.572328 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.572315    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-home\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:45:18.572328 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.572328    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-tmp-dir\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:45:18.574196 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.574160    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-dshm" (OuterVolumeSpecName: "dshm") pod "b2ab527b-de71-46ff-9368-46a0f74ab759" (UID: "b2ab527b-de71-46ff-9368-46a0f74ab759"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:45:18.574196 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.574170    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ab527b-de71-46ff-9368-46a0f74ab759-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2ab527b-de71-46ff-9368-46a0f74ab759" (UID: "b2ab527b-de71-46ff-9368-46a0f74ab759"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:45:18.574380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.574212    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ab527b-de71-46ff-9368-46a0f74ab759-kube-api-access-6qs4h" (OuterVolumeSpecName: "kube-api-access-6qs4h") pod "b2ab527b-de71-46ff-9368-46a0f74ab759" (UID: "b2ab527b-de71-46ff-9368-46a0f74ab759"). InnerVolumeSpecName "kube-api-access-6qs4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:45:18.673504 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.673467    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-model-cache\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:45:18.673504 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.673498    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ab527b-de71-46ff-9368-46a0f74ab759-dshm\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:45:18.673504 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.673509    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ab527b-de71-46ff-9368-46a0f74ab759-tls-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:45:18.673749 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:18.673518    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qs4h\" (UniqueName: \"kubernetes.io/projected/b2ab527b-de71-46ff-9368-46a0f74ab759-kube-api-access-6qs4h\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:45:19.330289 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:19.330252    2571 generic.go:356] "Generic (PLEG): container finished" podID="b2ab527b-de71-46ff-9368-46a0f74ab759" containerID="982b279d26e8c0a1a989bdad8ea9ac94fae2a09ce004d59dc84badca9bda3d96" exitCode=0
Sep 06 19:45:19.330677 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:19.330304    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"
Sep 06 19:45:19.330677 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:19.330329    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms" event={"ID":"b2ab527b-de71-46ff-9368-46a0f74ab759","Type":"ContainerDied","Data":"982b279d26e8c0a1a989bdad8ea9ac94fae2a09ce004d59dc84badca9bda3d96"}
Sep 06 19:45:19.330677 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:19.330377    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms" event={"ID":"b2ab527b-de71-46ff-9368-46a0f74ab759","Type":"ContainerDied","Data":"aeb665958bbbed5b5babb6dbd43576432c752230d551ce1185883e07b34ff18e"}
Sep 06 19:45:19.330677 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:19.330395    2571 scope.go:117] "RemoveContainer" containerID="982b279d26e8c0a1a989bdad8ea9ac94fae2a09ce004d59dc84badca9bda3d96"
Sep 06 19:45:19.339175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:19.339155    2571 scope.go:117] "RemoveContainer" containerID="982b279d26e8c0a1a989bdad8ea9ac94fae2a09ce004d59dc84badca9bda3d96"
Sep 06 19:45:19.339454 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:45:19.339436    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"982b279d26e8c0a1a989bdad8ea9ac94fae2a09ce004d59dc84badca9bda3d96\": container with ID starting with 982b279d26e8c0a1a989bdad8ea9ac94fae2a09ce004d59dc84badca9bda3d96 not found: ID does not exist" containerID="982b279d26e8c0a1a989bdad8ea9ac94fae2a09ce004d59dc84badca9bda3d96"
Sep 06 19:45:19.339489 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:19.339463    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"982b279d26e8c0a1a989bdad8ea9ac94fae2a09ce004d59dc84badca9bda3d96"} err="failed to get container status \"982b279d26e8c0a1a989bdad8ea9ac94fae2a09ce004d59dc84badca9bda3d96\": rpc error: code = NotFound desc = could not find container \"982b279d26e8c0a1a989bdad8ea9ac94fae2a09ce004d59dc84badca9bda3d96\": container with ID starting with 982b279d26e8c0a1a989bdad8ea9ac94fae2a09ce004d59dc84badca9bda3d96 not found: ID does not exist"
Sep 06 19:45:19.344880 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:19.344855    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"]
Sep 06 19:45:19.349084 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:19.349058    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-838def/explicit-model-838def-kserve-659899bf45-t8qms"]
Sep 06 19:45:20.252244 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:20.252209    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"]
Sep 06 19:45:20.252468 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:20.252442    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g" podUID="e3b29af8-0ce2-4d82-881e-ae3de5817fa3" containerName="istio-proxy" containerID="cri-o://253acdee6e6791e284cc0c97c5472aab4d3706d99736dc15a40cc1f53f1be74d" gracePeriod=30
Sep 06 19:45:20.919441 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:20.919406    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2ab527b-de71-46ff-9368-46a0f74ab759" path="/var/lib/kubelet/pods/b2ab527b-de71-46ff-9368-46a0f74ab759/volumes"
Sep 06 19:45:24.438774 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.438741    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"]
Sep 06 19:45:24.439273 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.439148    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="989c1c08-e461-4680-9743-96ecbcdd955e" containerName="cleanup"
Sep 06 19:45:24.439273 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.439160    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="989c1c08-e461-4680-9743-96ecbcdd955e" containerName="cleanup"
Sep 06 19:45:24.439273 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.439181    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ab527b-de71-46ff-9368-46a0f74ab759" containerName="main"
Sep 06 19:45:24.439273 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.439186    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ab527b-de71-46ff-9368-46a0f74ab759" containerName="main"
Sep 06 19:45:24.439273 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.439194    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27ad3d00-06f8-4cfd-8fbd-386f026f0175" containerName="cleanup"
Sep 06 19:45:24.439273 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.439200    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="27ad3d00-06f8-4cfd-8fbd-386f026f0175" containerName="cleanup"
Sep 06 19:45:24.439273 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.439273    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2ab527b-de71-46ff-9368-46a0f74ab759" containerName="main"
Sep 06 19:45:24.439655 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.439283    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="989c1c08-e461-4680-9743-96ecbcdd955e" containerName="cleanup"
Sep 06 19:45:24.439655 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.439291    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="27ad3d00-06f8-4cfd-8fbd-386f026f0175" containerName="cleanup"
Sep 06 19:45:24.443796 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.443767    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.446239 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.446215    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-eb3b3780-openshift-default-dockercfg-f6pkg\""
Sep 06 19:45:24.455377 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.455345    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"]
Sep 06 19:45:24.528975 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.528924    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-podinfo\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.529191 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.529013    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-envoy\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.529191 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.529067    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-token\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.529191 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.529092    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp225\" (UniqueName: \"kubernetes.io/projected/8e923128-5de4-47ad-bb8c-665680a6cbd4-kube-api-access-xp225\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.529191 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.529147    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e923128-5de4-47ad-bb8c-665680a6cbd4-istiod-ca-cert\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.529433 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.529199    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-workload-socket\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.529433 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.529252    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-credential-socket\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.529433 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.529332    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-data\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.529433 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.529361    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-workload-certs\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.631041 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.630992    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-credential-socket\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.631257 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.631084    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-data\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.631257 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.631113    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-workload-certs\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.631257 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.631191    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-podinfo\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.631257 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.631220    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-envoy\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.631481 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.631265    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-token\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.631481 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.631303    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp225\" (UniqueName: \"kubernetes.io/projected/8e923128-5de4-47ad-bb8c-665680a6cbd4-kube-api-access-xp225\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.631481 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.631354    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e923128-5de4-47ad-bb8c-665680a6cbd4-istiod-ca-cert\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.631481 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.631394    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-workload-socket\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.631708 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.631484    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-credential-socket\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.631708 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.631536    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-data\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.631859 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.631746    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-workload-socket\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.631859 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.631775    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-workload-certs\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.632137 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.632116    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e923128-5de4-47ad-bb8c-665680a6cbd4-istiod-ca-cert\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.633637 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.633607    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-envoy\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.633731 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.633707    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-podinfo\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.639280 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.639251    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-token\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.639389 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.639365    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp225\" (UniqueName: \"kubernetes.io/projected/8e923128-5de4-47ad-bb8c-665680a6cbd4-kube-api-access-xp225\") pod \"e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") " pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:24.757319 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:24.757214    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:25.036588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.036551    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"]
Sep 06 19:45:25.037925 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:45:25.037895    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e923128_5de4_47ad_bb8c_665680a6cbd4.slice/crio-5ab01b574689af17fe7dac1c6016459e0a498b49fbea35526549e39ac68680fc WatchSource:0}: Error finding container 5ab01b574689af17fe7dac1c6016459e0a498b49fbea35526549e39ac68680fc: Status 404 returned error can't find the container with id 5ab01b574689af17fe7dac1c6016459e0a498b49fbea35526549e39ac68680fc
Sep 06 19:45:25.040619 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.040577    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:45:25.040733 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.040670    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:45:25.040733 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.040713    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:45:25.374291 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.374258    2571 generic.go:356] "Generic (PLEG): container finished" podID="e3b29af8-0ce2-4d82-881e-ae3de5817fa3" containerID="253acdee6e6791e284cc0c97c5472aab4d3706d99736dc15a40cc1f53f1be74d" exitCode=0
Sep 06 19:45:25.374494 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.374327    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g" event={"ID":"e3b29af8-0ce2-4d82-881e-ae3de5817fa3","Type":"ContainerDied","Data":"253acdee6e6791e284cc0c97c5472aab4d3706d99736dc15a40cc1f53f1be74d"}
Sep 06 19:45:25.374494 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.374388    2571 scope.go:117] "RemoveContainer" containerID="25c10507f2581dc3ee96342e2d6afb103370ebb310cbb4b79a112d2811696109"
Sep 06 19:45:25.376151 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.376115    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" event={"ID":"8e923128-5de4-47ad-bb8c-665680a6cbd4","Type":"ContainerStarted","Data":"12c581a817fae33aaccf7b8d08984d7787905e88eb4a6c8bd9e05269169f737e"}
Sep 06 19:45:25.376262 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.376162    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" event={"ID":"8e923128-5de4-47ad-bb8c-665680a6cbd4","Type":"ContainerStarted","Data":"5ab01b574689af17fe7dac1c6016459e0a498b49fbea35526549e39ac68680fc"}
Sep 06 19:45:25.394223 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.394162    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" podStartSLOduration=1.394143612 podStartE2EDuration="1.394143612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:45:25.392485319 +0000 UTC m=+1088.962783665" watchObservedRunningTime="2026-09-06 19:45:25.394143612 +0000 UTC m=+1088.964442043"
Sep 06 19:45:25.512734 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.512704    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:45:25.541263 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.541222    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-token\") pod \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") "
Sep 06 19:45:25.541447 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.541281    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-envoy\") pod \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") "
Sep 06 19:45:25.541447 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.541356    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-data\") pod \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") "
Sep 06 19:45:25.541447 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.541397    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-credential-socket\") pod \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") "
Sep 06 19:45:25.541447 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.541438    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-podinfo\") pod \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") "
Sep 06 19:45:25.541649 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.541471    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-workload-certs\") pod \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") "
Sep 06 19:45:25.541649 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.541525    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8cn2\" (UniqueName: \"kubernetes.io/projected/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-kube-api-access-d8cn2\") pod \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") "
Sep 06 19:45:25.541649 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.541552    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-workload-socket\") pod \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") "
Sep 06 19:45:25.541649 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.541597    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istiod-ca-cert\") pod \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\" (UID: \"e3b29af8-0ce2-4d82-881e-ae3de5817fa3\") "
Sep 06 19:45:25.541920 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.541688    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3b29af8-0ce2-4d82-881e-ae3de5817fa3" (UID: "e3b29af8-0ce2-4d82-881e-ae3de5817fa3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:45:25.541920 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.541842    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3b29af8-0ce2-4d82-881e-ae3de5817fa3" (UID: "e3b29af8-0ce2-4d82-881e-ae3de5817fa3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:45:25.542064 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.541930    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3b29af8-0ce2-4d82-881e-ae3de5817fa3" (UID: "e3b29af8-0ce2-4d82-881e-ae3de5817fa3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:45:25.542129 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.542060    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3b29af8-0ce2-4d82-881e-ae3de5817fa3" (UID: "e3b29af8-0ce2-4d82-881e-ae3de5817fa3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:45:25.542129 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.542089    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-data\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:45:25.542129 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.542110    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-credential-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:45:25.542129 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.542125    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-workload-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:45:25.542351 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.542132    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3b29af8-0ce2-4d82-881e-ae3de5817fa3" (UID: "e3b29af8-0ce2-4d82-881e-ae3de5817fa3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:45:25.543693 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.543659    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3b29af8-0ce2-4d82-881e-ae3de5817fa3" (UID: "e3b29af8-0ce2-4d82-881e-ae3de5817fa3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:45:25.544019 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.543988    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3b29af8-0ce2-4d82-881e-ae3de5817fa3" (UID: "e3b29af8-0ce2-4d82-881e-ae3de5817fa3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 19:45:25.544120 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.544078    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3b29af8-0ce2-4d82-881e-ae3de5817fa3" (UID: "e3b29af8-0ce2-4d82-881e-ae3de5817fa3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:45:25.544173 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.544109    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-kube-api-access-d8cn2" (OuterVolumeSpecName: "kube-api-access-d8cn2") pod "e3b29af8-0ce2-4d82-881e-ae3de5817fa3" (UID: "e3b29af8-0ce2-4d82-881e-ae3de5817fa3"). InnerVolumeSpecName "kube-api-access-d8cn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:45:25.643561 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.643530    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-podinfo\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:45:25.643561 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.643561    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8cn2\" (UniqueName: \"kubernetes.io/projected/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-kube-api-access-d8cn2\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:45:25.643737 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.643570    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-workload-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:45:25.643737 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.643580    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istiod-ca-cert\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:45:25.643737 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.643588    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-token\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:45:25.643737 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.643595    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3b29af8-0ce2-4d82-881e-ae3de5817fa3-istio-envoy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:45:25.757859 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:25.757799    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:26.135148 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:26.135109    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:45:26.382938 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:26.382907    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"
Sep 06 19:45:26.383123 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:26.382951    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g" event={"ID":"e3b29af8-0ce2-4d82-881e-ae3de5817fa3","Type":"ContainerDied","Data":"48c723c9ba32bc401d482fc0f3d6ee442a4cb7066bfd0d9a94ef2ce7a0e82835"}
Sep 06 19:45:26.383123 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:26.383030    2571 scope.go:117] "RemoveContainer" containerID="253acdee6e6791e284cc0c97c5472aab4d3706d99736dc15a40cc1f53f1be74d"
Sep 06 19:45:26.404701 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:26.404663    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"]
Sep 06 19:45:26.408430 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:26.408391    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-838def-openshift-default-54444657cb-flr2g"]
Sep 06 19:45:26.763733 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:26.763654    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:26.922406 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:26.922373    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3b29af8-0ce2-4d82-881e-ae3de5817fa3" path="/var/lib/kubelet/pods/e3b29af8-0ce2-4d82-881e-ae3de5817fa3/volumes"
Sep 06 19:45:27.390413 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:27.390385    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:27.391485 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:27.391461    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:39.641435 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:39.641390    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:45:43.432180 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:43.432133    2571 patch_prober.go:28] interesting pod/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 06 19:45:43.432593 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:43.432205    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 06 19:45:44.492072 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:44.492038    2571 generic.go:356] "Generic (PLEG): container finished" podID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerID="12c581a817fae33aaccf7b8d08984d7787905e88eb4a6c8bd9e05269169f737e" exitCode=0
Sep 06 19:45:44.492514 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:44.492112    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" event={"ID":"8e923128-5de4-47ad-bb8c-665680a6cbd4","Type":"ContainerDied","Data":"12c581a817fae33aaccf7b8d08984d7787905e88eb4a6c8bd9e05269169f737e"}
Sep 06 19:45:44.492561 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:44.492514    2571 scope.go:117] "RemoveContainer" containerID="12c581a817fae33aaccf7b8d08984d7787905e88eb4a6c8bd9e05269169f737e"
Sep 06 19:45:44.493642 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:44.493614    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:45:44.493707 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:44.493685    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:45:44.493745 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:44.493719    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:45:45.499195 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:45.499158    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" event={"ID":"8e923128-5de4-47ad-bb8c-665680a6cbd4","Type":"ContainerStarted","Data":"f8456d43249f71280e4fea4c399f45d73a7a91e6bcde99f23144f65bc928378b"}
Sep 06 19:45:45.758634 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:45.758536    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:45.759660 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:45.758905    2571 patch_prober.go:28] interesting pod/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 06 19:45:45.759903 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:45.759876    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 06 19:45:46.757981 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:46.757913    2571 patch_prober.go:28] interesting pod/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 06 19:45:46.758405 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:46.758021    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 06 19:45:47.555528 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.555492    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"]
Sep 06 19:45:47.564540 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.564505    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b29af8-0ce2-4d82-881e-ae3de5817fa3" containerName="istio-proxy"
Sep 06 19:45:47.564784 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.564766    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b29af8-0ce2-4d82-881e-ae3de5817fa3" containerName="istio-proxy"
Sep 06 19:45:47.564901 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.564889    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b29af8-0ce2-4d82-881e-ae3de5817fa3" containerName="istio-proxy"
Sep 06 19:45:47.565015 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.565004    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b29af8-0ce2-4d82-881e-ae3de5817fa3" containerName="istio-proxy"
Sep 06 19:45:47.565271 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.565255    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b29af8-0ce2-4d82-881e-ae3de5817fa3" containerName="istio-proxy"
Sep 06 19:45:47.565408 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.565396    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b29af8-0ce2-4d82-881e-ae3de5817fa3" containerName="istio-proxy"
Sep 06 19:45:47.569253 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.569221    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"]
Sep 06 19:45:47.569542 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.569522    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.573813 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.573788    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2c0c66\"/\"default-dockercfg-h54fh\""
Sep 06 19:45:47.574746 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.573804    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2c0c66\"/\"noten-model-2c0c66-kserve-self-signed-certs\""
Sep 06 19:45:47.575039 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.573924    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2c0c66\"/\"kube-root-ca.crt\""
Sep 06 19:45:47.575213 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.573988    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2c0c66\"/\"openshift-service-ca.crt\""
Sep 06 19:45:47.679533 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.679492    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5xzd\" (UniqueName: \"kubernetes.io/projected/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-kube-api-access-m5xzd\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.679722 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.679559    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-home\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.679722 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.679606    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-tls-certs\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.679722 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.679657    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-dshm\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.679722 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.679674    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-model-cache\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.679873 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.679752    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-tmp-dir\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.759207 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.758675    2571 patch_prober.go:28] interesting pod/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 06 19:45:47.759207 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.758742    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 06 19:45:47.780930 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.780887    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-home\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.780930 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.780929    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-tls-certs\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.781233 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.780951    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-dshm\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.781233 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.781040    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-model-cache\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.781233 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.781174    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-tmp-dir\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.781402 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.781243    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xzd\" (UniqueName: \"kubernetes.io/projected/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-kube-api-access-m5xzd\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.781461 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.781401    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-home\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.781461 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.781436    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-model-cache\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.781603 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.781571    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-tmp-dir\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.783284 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.783260    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-dshm\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.783553 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.783536    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-tls-certs\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.788625 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.788602    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xzd\" (UniqueName: \"kubernetes.io/projected/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-kube-api-access-m5xzd\") pod \"noten-model-2c0c66-kserve-f9b75655f-rkln7\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") " pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:47.882840 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:47.882796    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:45:48.175784 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:48.175755    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"]
Sep 06 19:45:48.177588 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:45:48.177556    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3573b41_2c7f_4457_9f4b_c22432b9bf1c.slice/crio-2274ecb010f2ddf24362d9ac37ed56652c20475a75241dad71492ff44098f50c WatchSource:0}: Error finding container 2274ecb010f2ddf24362d9ac37ed56652c20475a75241dad71492ff44098f50c: Status 404 returned error can't find the container with id 2274ecb010f2ddf24362d9ac37ed56652c20475a75241dad71492ff44098f50c
Sep 06 19:45:48.516481 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:48.516371    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7" event={"ID":"f3573b41-2c7f-4457-9f4b-c22432b9bf1c","Type":"ContainerStarted","Data":"51722266ecdf95bf34cdfc1ed59e1b6f974b6d8955b3f2bfe598c19d4549aaff"}
Sep 06 19:45:48.516481 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:48.516423    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7" event={"ID":"f3573b41-2c7f-4457-9f4b-c22432b9bf1c","Type":"ContainerStarted","Data":"2274ecb010f2ddf24362d9ac37ed56652c20475a75241dad71492ff44098f50c"}
Sep 06 19:45:48.532254 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:48.532200    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7" podStartSLOduration=1.532181493 podStartE2EDuration="1.532181493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:45:48.530391308 +0000 UTC m=+1112.100689655" watchObservedRunningTime="2026-09-06 19:45:48.532181493 +0000 UTC m=+1112.102479839"
Sep 06 19:45:48.758331 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:48.758292    2571 patch_prober.go:28] interesting pod/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 06 19:45:48.758542 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:48.758360    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 06 19:45:49.651182 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:49.650359    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"]
Sep 06 19:45:49.758763 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:49.758709    2571 patch_prober.go:28] interesting pod/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 06 19:45:49.759000 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:49.758783    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 06 19:45:50.624281 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:50.624234    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"]
Sep 06 19:45:50.758500 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:50.758459    2571 patch_prober.go:28] interesting pod/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 06 19:45:50.759090 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:50.759054    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 06 19:45:51.761594 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:51.761557    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:51.762073 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:51.761855    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:51.762595 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:51.762579    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:45:51.929373 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:51.929336    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"]
Sep 06 19:45:52.043204 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:52.043111    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:45:57.883094 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:45:57.883056    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:46:02.745108 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:02.745073    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6433a6-29812065-rkj4x"]
Sep 06 19:46:02.746887 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:02.746864    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6433a6-29812065-rkj4x"]
Sep 06 19:46:02.918623 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:02.918581    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="989c1c08-e461-4680-9743-96ecbcdd955e" path="/var/lib/kubelet/pods/989c1c08-e461-4680-9743-96ecbcdd955e/volumes"
Sep 06 19:46:04.675492 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:04.675426    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc" podUID="e322d7ba-9018-40e9-914d-8e4ca2d6ca0e" containerName="main" containerID="cri-o://714b41238b5511e46e7a462f5aa13305654d50a56bd2472c15b8b6cfc32ddabf" gracePeriod=45
Sep 06 19:46:04.947666 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:04.947641    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:46:05.035883 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.035843    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-tls-certs\") pod \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") "
Sep 06 19:46:05.036110 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.035923    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-dshm\") pod \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") "
Sep 06 19:46:05.036110 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.035948    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s47fv\" (UniqueName: \"kubernetes.io/projected/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-kube-api-access-s47fv\") pod \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") "
Sep 06 19:46:05.036110 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.036097    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-model-cache\") pod \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") "
Sep 06 19:46:05.036258 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.036133    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-tmp-dir\") pod \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") "
Sep 06 19:46:05.036258 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.036172    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-home\") pod \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\" (UID: \"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e\") "
Sep 06 19:46:05.036375 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.036354    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e322d7ba-9018-40e9-914d-8e4ca2d6ca0e" (UID: "e322d7ba-9018-40e9-914d-8e4ca2d6ca0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:05.036450 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.036398    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e322d7ba-9018-40e9-914d-8e4ca2d6ca0e" (UID: "e322d7ba-9018-40e9-914d-8e4ca2d6ca0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:05.036500 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.036466    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-home" (OuterVolumeSpecName: "home") pod "e322d7ba-9018-40e9-914d-8e4ca2d6ca0e" (UID: "e322d7ba-9018-40e9-914d-8e4ca2d6ca0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:05.036641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.036623    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-model-cache\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:05.036690 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.036649    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-tmp-dir\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:05.036690 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.036663    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-home\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:05.038345 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.038324    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-kube-api-access-s47fv" (OuterVolumeSpecName: "kube-api-access-s47fv") pod "e322d7ba-9018-40e9-914d-8e4ca2d6ca0e" (UID: "e322d7ba-9018-40e9-914d-8e4ca2d6ca0e"). InnerVolumeSpecName "kube-api-access-s47fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:46:05.038626 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.038602    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e322d7ba-9018-40e9-914d-8e4ca2d6ca0e" (UID: "e322d7ba-9018-40e9-914d-8e4ca2d6ca0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:46:05.038626 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.038609    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-dshm" (OuterVolumeSpecName: "dshm") pod "e322d7ba-9018-40e9-914d-8e4ca2d6ca0e" (UID: "e322d7ba-9018-40e9-914d-8e4ca2d6ca0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:05.138012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.137950    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-dshm\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:05.138012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.138009    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s47fv\" (UniqueName: \"kubernetes.io/projected/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-kube-api-access-s47fv\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:05.138012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.138021    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e-tls-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:05.535261 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.535226    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:46:05.611468 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.611432    2571 generic.go:356] "Generic (PLEG): container finished" podID="e322d7ba-9018-40e9-914d-8e4ca2d6ca0e" containerID="714b41238b5511e46e7a462f5aa13305654d50a56bd2472c15b8b6cfc32ddabf" exitCode=0
Sep 06 19:46:05.611685 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.611486    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"
Sep 06 19:46:05.611685 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.611477    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc" event={"ID":"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e","Type":"ContainerDied","Data":"714b41238b5511e46e7a462f5aa13305654d50a56bd2472c15b8b6cfc32ddabf"}
Sep 06 19:46:05.611685 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.611614    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc" event={"ID":"e322d7ba-9018-40e9-914d-8e4ca2d6ca0e","Type":"ContainerDied","Data":"89f6deb2bd24dd9ec5f285f2b887101b488d2934462cdd869cd97466e43667b3"}
Sep 06 19:46:05.611685 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.611639    2571 scope.go:117] "RemoveContainer" containerID="714b41238b5511e46e7a462f5aa13305654d50a56bd2472c15b8b6cfc32ddabf"
Sep 06 19:46:05.623069 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.622952    2571 scope.go:117] "RemoveContainer" containerID="714b41238b5511e46e7a462f5aa13305654d50a56bd2472c15b8b6cfc32ddabf"
Sep 06 19:46:05.623479 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:46:05.623425    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714b41238b5511e46e7a462f5aa13305654d50a56bd2472c15b8b6cfc32ddabf\": container with ID starting with 714b41238b5511e46e7a462f5aa13305654d50a56bd2472c15b8b6cfc32ddabf not found: ID does not exist" containerID="714b41238b5511e46e7a462f5aa13305654d50a56bd2472c15b8b6cfc32ddabf"
Sep 06 19:46:05.623610 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.623495    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714b41238b5511e46e7a462f5aa13305654d50a56bd2472c15b8b6cfc32ddabf"} err="failed to get container status \"714b41238b5511e46e7a462f5aa13305654d50a56bd2472c15b8b6cfc32ddabf\": rpc error: code = NotFound desc = could not find container \"714b41238b5511e46e7a462f5aa13305654d50a56bd2472c15b8b6cfc32ddabf\": container with ID starting with 714b41238b5511e46e7a462f5aa13305654d50a56bd2472c15b8b6cfc32ddabf not found: ID does not exist"
Sep 06 19:46:05.635731 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.635695    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"]
Sep 06 19:46:05.638860 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.638830    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6433a6/test-model-6433a6-kserve-656666fd6c-lpttc"]
Sep 06 19:46:05.680185 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.680141    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7" podUID="f3573b41-2c7f-4457-9f4b-c22432b9bf1c" containerName="main" containerID="cri-o://51722266ecdf95bf34cdfc1ed59e1b6f974b6d8955b3f2bfe598c19d4549aaff" gracePeriod=45
Sep 06 19:46:05.937203 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:05.937171    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:46:06.046783 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.046740    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-dshm\") pod \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") "
Sep 06 19:46:06.046946 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.046800    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-home\") pod \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") "
Sep 06 19:46:06.046946 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.046834    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-tls-certs\") pod \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") "
Sep 06 19:46:06.046946 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.046927    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5xzd\" (UniqueName: \"kubernetes.io/projected/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-kube-api-access-m5xzd\") pod \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") "
Sep 06 19:46:06.047198 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.046976    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-model-cache\") pod \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") "
Sep 06 19:46:06.047198 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.047018    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-tmp-dir\") pod \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\" (UID: \"f3573b41-2c7f-4457-9f4b-c22432b9bf1c\") "
Sep 06 19:46:06.047198 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.047109    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-home" (OuterVolumeSpecName: "home") pod "f3573b41-2c7f-4457-9f4b-c22432b9bf1c" (UID: "f3573b41-2c7f-4457-9f4b-c22432b9bf1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:06.047373 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.047314    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3573b41-2c7f-4457-9f4b-c22432b9bf1c" (UID: "f3573b41-2c7f-4457-9f4b-c22432b9bf1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:06.047437 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.047395    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3573b41-2c7f-4457-9f4b-c22432b9bf1c" (UID: "f3573b41-2c7f-4457-9f4b-c22432b9bf1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:06.047489 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.047439    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-home\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:06.047489 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.047456    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-model-cache\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:06.049176 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.049146    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-dshm" (OuterVolumeSpecName: "dshm") pod "f3573b41-2c7f-4457-9f4b-c22432b9bf1c" (UID: "f3573b41-2c7f-4457-9f4b-c22432b9bf1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:06.049176 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.049159    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3573b41-2c7f-4457-9f4b-c22432b9bf1c" (UID: "f3573b41-2c7f-4457-9f4b-c22432b9bf1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:46:06.049353 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.049273    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-kube-api-access-m5xzd" (OuterVolumeSpecName: "kube-api-access-m5xzd") pod "f3573b41-2c7f-4457-9f4b-c22432b9bf1c" (UID: "f3573b41-2c7f-4457-9f4b-c22432b9bf1c"). InnerVolumeSpecName "kube-api-access-m5xzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:46:06.148460 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.148412    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-tmp-dir\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:06.148460 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.148449    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-dshm\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:06.148460 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.148457    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-tls-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:06.148460 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.148470    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5xzd\" (UniqueName: \"kubernetes.io/projected/f3573b41-2c7f-4457-9f4b-c22432b9bf1c-kube-api-access-m5xzd\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:06.619158 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.619119    2571 generic.go:356] "Generic (PLEG): container finished" podID="f3573b41-2c7f-4457-9f4b-c22432b9bf1c" containerID="51722266ecdf95bf34cdfc1ed59e1b6f974b6d8955b3f2bfe598c19d4549aaff" exitCode=0
Sep 06 19:46:06.619367 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.619165    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"
Sep 06 19:46:06.619367 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.619186    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7" event={"ID":"f3573b41-2c7f-4457-9f4b-c22432b9bf1c","Type":"ContainerDied","Data":"51722266ecdf95bf34cdfc1ed59e1b6f974b6d8955b3f2bfe598c19d4549aaff"}
Sep 06 19:46:06.619367 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.619223    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7" event={"ID":"f3573b41-2c7f-4457-9f4b-c22432b9bf1c","Type":"ContainerDied","Data":"2274ecb010f2ddf24362d9ac37ed56652c20475a75241dad71492ff44098f50c"}
Sep 06 19:46:06.619367 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.619245    2571 scope.go:117] "RemoveContainer" containerID="51722266ecdf95bf34cdfc1ed59e1b6f974b6d8955b3f2bfe598c19d4549aaff"
Sep 06 19:46:06.629180 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.629158    2571 scope.go:117] "RemoveContainer" containerID="51722266ecdf95bf34cdfc1ed59e1b6f974b6d8955b3f2bfe598c19d4549aaff"
Sep 06 19:46:06.629494 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:46:06.629469    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51722266ecdf95bf34cdfc1ed59e1b6f974b6d8955b3f2bfe598c19d4549aaff\": container with ID starting with 51722266ecdf95bf34cdfc1ed59e1b6f974b6d8955b3f2bfe598c19d4549aaff not found: ID does not exist" containerID="51722266ecdf95bf34cdfc1ed59e1b6f974b6d8955b3f2bfe598c19d4549aaff"
Sep 06 19:46:06.629563 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.629502    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51722266ecdf95bf34cdfc1ed59e1b6f974b6d8955b3f2bfe598c19d4549aaff"} err="failed to get container status \"51722266ecdf95bf34cdfc1ed59e1b6f974b6d8955b3f2bfe598c19d4549aaff\": rpc error: code = NotFound desc = could not find container \"51722266ecdf95bf34cdfc1ed59e1b6f974b6d8955b3f2bfe598c19d4549aaff\": container with ID starting with 51722266ecdf95bf34cdfc1ed59e1b6f974b6d8955b3f2bfe598c19d4549aaff not found: ID does not exist"
Sep 06 19:46:06.640825 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.640794    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"]
Sep 06 19:46:06.644477 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.644448    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-2c0c66/noten-model-2c0c66-kserve-f9b75655f-rkln7"]
Sep 06 19:46:06.919409 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.919311    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e322d7ba-9018-40e9-914d-8e4ca2d6ca0e" path="/var/lib/kubelet/pods/e322d7ba-9018-40e9-914d-8e4ca2d6ca0e/volumes"
Sep 06 19:46:06.919935 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.919914    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3573b41-2c7f-4457-9f4b-c22432b9bf1c" path="/var/lib/kubelet/pods/f3573b41-2c7f-4457-9f4b-c22432b9bf1c/volumes"
Sep 06 19:46:06.950302 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:06.950256    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb" podUID="ae277b9f-bca2-4a4b-9da5-7fba9160acee" containerName="main" containerID="cri-o://518b7f01ee5c1dc313e8f7364d49d2c69f44f7b81c43ba0b7fa568382d5e0dd0" gracePeriod=45
Sep 06 19:46:07.204620 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.204590    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:46:07.256762 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.256732    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-model-cache\") pod \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") "
Sep 06 19:46:07.256979 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.256769    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-home\") pod \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") "
Sep 06 19:46:07.256979 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.256805    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-tmp-dir\") pod \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") "
Sep 06 19:46:07.256979 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.256916    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-dshm\") pod \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") "
Sep 06 19:46:07.257177 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.256993    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae277b9f-bca2-4a4b-9da5-7fba9160acee" (UID: "ae277b9f-bca2-4a4b-9da5-7fba9160acee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:07.257177 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.257010    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-home" (OuterVolumeSpecName: "home") pod "ae277b9f-bca2-4a4b-9da5-7fba9160acee" (UID: "ae277b9f-bca2-4a4b-9da5-7fba9160acee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:07.257177 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.257018    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae277b9f-bca2-4a4b-9da5-7fba9160acee-tls-certs\") pod \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") "
Sep 06 19:46:07.257177 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.257076    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae277b9f-bca2-4a4b-9da5-7fba9160acee" (UID: "ae277b9f-bca2-4a4b-9da5-7fba9160acee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:07.257177 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.257121    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b4c9\" (UniqueName: \"kubernetes.io/projected/ae277b9f-bca2-4a4b-9da5-7fba9160acee-kube-api-access-4b4c9\") pod \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\" (UID: \"ae277b9f-bca2-4a4b-9da5-7fba9160acee\") "
Sep 06 19:46:07.257440 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.257423    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-model-cache\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:07.257499 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.257441    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-home\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:07.257499 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.257449    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-tmp-dir\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:07.259156 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.259132    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-dshm" (OuterVolumeSpecName: "dshm") pod "ae277b9f-bca2-4a4b-9da5-7fba9160acee" (UID: "ae277b9f-bca2-4a4b-9da5-7fba9160acee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:07.259156 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.259151    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae277b9f-bca2-4a4b-9da5-7fba9160acee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae277b9f-bca2-4a4b-9da5-7fba9160acee" (UID: "ae277b9f-bca2-4a4b-9da5-7fba9160acee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:46:07.259317 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.259284    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae277b9f-bca2-4a4b-9da5-7fba9160acee-kube-api-access-4b4c9" (OuterVolumeSpecName: "kube-api-access-4b4c9") pod "ae277b9f-bca2-4a4b-9da5-7fba9160acee" (UID: "ae277b9f-bca2-4a4b-9da5-7fba9160acee"). InnerVolumeSpecName "kube-api-access-4b4c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:46:07.358884 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.358831    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae277b9f-bca2-4a4b-9da5-7fba9160acee-tls-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:07.358884 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.358877    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b4c9\" (UniqueName: \"kubernetes.io/projected/ae277b9f-bca2-4a4b-9da5-7fba9160acee-kube-api-access-4b4c9\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:07.358884 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.358889    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae277b9f-bca2-4a4b-9da5-7fba9160acee-dshm\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:07.626086 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.626011    2571 generic.go:356] "Generic (PLEG): container finished" podID="ae277b9f-bca2-4a4b-9da5-7fba9160acee" containerID="518b7f01ee5c1dc313e8f7364d49d2c69f44f7b81c43ba0b7fa568382d5e0dd0" exitCode=0
Sep 06 19:46:07.626086 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.626058    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb" event={"ID":"ae277b9f-bca2-4a4b-9da5-7fba9160acee","Type":"ContainerDied","Data":"518b7f01ee5c1dc313e8f7364d49d2c69f44f7b81c43ba0b7fa568382d5e0dd0"}
Sep 06 19:46:07.626086 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.626085    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"
Sep 06 19:46:07.626376 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.626114    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb" event={"ID":"ae277b9f-bca2-4a4b-9da5-7fba9160acee","Type":"ContainerDied","Data":"d4446f4bdcc7593f2260ecc555d0c4e7ebc7cea486c434bad03202756ce62068"}
Sep 06 19:46:07.626376 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.626138    2571 scope.go:117] "RemoveContainer" containerID="518b7f01ee5c1dc313e8f7364d49d2c69f44f7b81c43ba0b7fa568382d5e0dd0"
Sep 06 19:46:07.637265 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.637242    2571 scope.go:117] "RemoveContainer" containerID="518b7f01ee5c1dc313e8f7364d49d2c69f44f7b81c43ba0b7fa568382d5e0dd0"
Sep 06 19:46:07.637599 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:46:07.637577    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518b7f01ee5c1dc313e8f7364d49d2c69f44f7b81c43ba0b7fa568382d5e0dd0\": container with ID starting with 518b7f01ee5c1dc313e8f7364d49d2c69f44f7b81c43ba0b7fa568382d5e0dd0 not found: ID does not exist" containerID="518b7f01ee5c1dc313e8f7364d49d2c69f44f7b81c43ba0b7fa568382d5e0dd0"
Sep 06 19:46:07.637640 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.637609    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518b7f01ee5c1dc313e8f7364d49d2c69f44f7b81c43ba0b7fa568382d5e0dd0"} err="failed to get container status \"518b7f01ee5c1dc313e8f7364d49d2c69f44f7b81c43ba0b7fa568382d5e0dd0\": rpc error: code = NotFound desc = could not find container \"518b7f01ee5c1dc313e8f7364d49d2c69f44f7b81c43ba0b7fa568382d5e0dd0\": container with ID starting with 518b7f01ee5c1dc313e8f7364d49d2c69f44f7b81c43ba0b7fa568382d5e0dd0 not found: ID does not exist"
Sep 06 19:46:07.648478 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.648443    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"]
Sep 06 19:46:07.652156 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:07.652125    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d5bb2d/test-model-d5bb2d-kserve-bf5b7b75-dmxmb"]
Sep 06 19:46:08.918666 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:08.918629    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae277b9f-bca2-4a4b-9da5-7fba9160acee" path="/var/lib/kubelet/pods/ae277b9f-bca2-4a4b-9da5-7fba9160acee/volumes"
Sep 06 19:46:09.642485 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.642442    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg"]
Sep 06 19:46:09.642848 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.642836    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae277b9f-bca2-4a4b-9da5-7fba9160acee" containerName="main"
Sep 06 19:46:09.642892 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.642850    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae277b9f-bca2-4a4b-9da5-7fba9160acee" containerName="main"
Sep 06 19:46:09.642892 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.642866    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e322d7ba-9018-40e9-914d-8e4ca2d6ca0e" containerName="main"
Sep 06 19:46:09.642892 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.642872    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e322d7ba-9018-40e9-914d-8e4ca2d6ca0e" containerName="main"
Sep 06 19:46:09.642892 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.642882    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3573b41-2c7f-4457-9f4b-c22432b9bf1c" containerName="main"
Sep 06 19:46:09.642892 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.642887    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3573b41-2c7f-4457-9f4b-c22432b9bf1c" containerName="main"
Sep 06 19:46:09.643075 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.642945    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e322d7ba-9018-40e9-914d-8e4ca2d6ca0e" containerName="main"
Sep 06 19:46:09.643075 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.642968    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3573b41-2c7f-4457-9f4b-c22432b9bf1c" containerName="main"
Sep 06 19:46:09.643075 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.642977    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae277b9f-bca2-4a4b-9da5-7fba9160acee" containerName="main"
Sep 06 19:46:09.647397 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.647377    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg"
Sep 06 19:46:09.649791 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.649766    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 19:46:09.649977 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.649794    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h45ww\""
Sep 06 19:46:09.653118 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.653088    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg"]
Sep 06 19:46:09.680732 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.680693    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg\" (UID: \"7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg"
Sep 06 19:46:09.781289 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.781255    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg\" (UID: \"7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg"
Sep 06 19:46:09.781906 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.781883    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg\" (UID: \"7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg"
Sep 06 19:46:09.959942 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:09.959839    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg"
Sep 06 19:46:10.230569 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:10.230505    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg"]
Sep 06 19:46:10.232987 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:46:10.232942    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7afaaee8_08a7_4af7_9cec_89fcd0eaf9c2.slice/crio-2220e2f9227749439c13cb3314d8118bfda64967f21fda0869bcdbb4b83d9057 WatchSource:0}: Error finding container 2220e2f9227749439c13cb3314d8118bfda64967f21fda0869bcdbb4b83d9057: Status 404 returned error can't find the container with id 2220e2f9227749439c13cb3314d8118bfda64967f21fda0869bcdbb4b83d9057
Sep 06 19:46:10.649007 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:10.648969    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg" event={"ID":"7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2","Type":"ContainerStarted","Data":"c0dfff963c2e4078f59763df50a0c1a04b0c9b9f6ad52cfe3ffcf256e4dd042e"}
Sep 06 19:46:10.649168 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:10.649031    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg" event={"ID":"7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2","Type":"ContainerStarted","Data":"2220e2f9227749439c13cb3314d8118bfda64967f21fda0869bcdbb4b83d9057"}
Sep 06 19:46:10.664077 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:10.664022    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg" podStartSLOduration=1.6640065229999998 podStartE2EDuration="1.664006523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:46:10.661541374 +0000 UTC m=+1134.231839720" watchObservedRunningTime="2026-09-06 19:46:10.664006523 +0000 UTC m=+1134.234304869"
Sep 06 19:46:11.655493 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:11.655458    2571 generic.go:356] "Generic (PLEG): container finished" podID="7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2" containerID="c0dfff963c2e4078f59763df50a0c1a04b0c9b9f6ad52cfe3ffcf256e4dd042e" exitCode=0
Sep 06 19:46:11.655887 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:11.655504    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg" event={"ID":"7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2","Type":"ContainerDied","Data":"c0dfff963c2e4078f59763df50a0c1a04b0c9b9f6ad52cfe3ffcf256e4dd042e"}
Sep 06 19:46:12.791923 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:12.791893    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg"
Sep 06 19:46:12.913485 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:12.913380    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2-maas-api-service-ca\") pod \"7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2\" (UID: \"7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2\") "
Sep 06 19:46:12.913743 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:12.913714    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2" (UID: "7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:46:13.014764 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:13.014725    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2-maas-api-service-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:13.667213 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:13.667186    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg"
Sep 06 19:46:13.667399 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:13.667220    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg" event={"ID":"7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2","Type":"ContainerDied","Data":"2220e2f9227749439c13cb3314d8118bfda64967f21fda0869bcdbb4b83d9057"}
Sep 06 19:46:13.667399 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:13.667273    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2220e2f9227749439c13cb3314d8118bfda64967f21fda0869bcdbb4b83d9057"
Sep 06 19:46:17.161681 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:17.161640    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"]
Sep 06 19:46:17.162199 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:17.161884    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerName="istio-proxy" containerID="cri-o://f8456d43249f71280e4fea4c399f45d73a7a91e6bcde99f23144f65bc928378b" gracePeriod=30
Sep 06 19:46:17.441741 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:17.441702    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:46:20.191232 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:20.191189    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg"]
Sep 06 19:46:20.193492 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:20.193462    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d5bb2d-p4gwg"]
Sep 06 19:46:20.918249 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:20.918213    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2" path="/var/lib/kubelet/pods/7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2/volumes"
Sep 06 19:46:21.764476 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:21.764431    2571 patch_prober.go:28] interesting pod/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 06 19:46:21.764856 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:21.764498    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 19:46:22.415309 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.415279    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:46:22.512250 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.512158    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-podinfo\") pod \"8e923128-5de4-47ad-bb8c-665680a6cbd4\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") "
Sep 06 19:46:22.512250 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.512202    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp225\" (UniqueName: \"kubernetes.io/projected/8e923128-5de4-47ad-bb8c-665680a6cbd4-kube-api-access-xp225\") pod \"8e923128-5de4-47ad-bb8c-665680a6cbd4\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") "
Sep 06 19:46:22.512250 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.512234    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-workload-certs\") pod \"8e923128-5de4-47ad-bb8c-665680a6cbd4\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") "
Sep 06 19:46:22.512519 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.512269    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-token\") pod \"8e923128-5de4-47ad-bb8c-665680a6cbd4\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") "
Sep 06 19:46:22.512519 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.512288    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e923128-5de4-47ad-bb8c-665680a6cbd4-istiod-ca-cert\") pod \"8e923128-5de4-47ad-bb8c-665680a6cbd4\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") "
Sep 06 19:46:22.512519 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.512315    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-workload-socket\") pod \"8e923128-5de4-47ad-bb8c-665680a6cbd4\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") "
Sep 06 19:46:22.512519 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.512340    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-data\") pod \"8e923128-5de4-47ad-bb8c-665680a6cbd4\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") "
Sep 06 19:46:22.512519 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.512399    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-credential-socket\") pod \"8e923128-5de4-47ad-bb8c-665680a6cbd4\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") "
Sep 06 19:46:22.512519 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.512432    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-envoy\") pod \"8e923128-5de4-47ad-bb8c-665680a6cbd4\" (UID: \"8e923128-5de4-47ad-bb8c-665680a6cbd4\") "
Sep 06 19:46:22.512777 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.512545    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e923128-5de4-47ad-bb8c-665680a6cbd4" (UID: "8e923128-5de4-47ad-bb8c-665680a6cbd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:22.512777 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.512724    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e923128-5de4-47ad-bb8c-665680a6cbd4" (UID: "8e923128-5de4-47ad-bb8c-665680a6cbd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:22.512900 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.512774    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e923128-5de4-47ad-bb8c-665680a6cbd4" (UID: "8e923128-5de4-47ad-bb8c-665680a6cbd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:22.512900 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.512795    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-workload-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:22.513030 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.513002    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e923128-5de4-47ad-bb8c-665680a6cbd4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e923128-5de4-47ad-bb8c-665680a6cbd4" (UID: "8e923128-5de4-47ad-bb8c-665680a6cbd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:46:22.513094 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.513029    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e923128-5de4-47ad-bb8c-665680a6cbd4" (UID: "8e923128-5de4-47ad-bb8c-665680a6cbd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:22.514587 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.514561    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e923128-5de4-47ad-bb8c-665680a6cbd4-kube-api-access-xp225" (OuterVolumeSpecName: "kube-api-access-xp225") pod "8e923128-5de4-47ad-bb8c-665680a6cbd4" (UID: "8e923128-5de4-47ad-bb8c-665680a6cbd4"). InnerVolumeSpecName "kube-api-access-xp225". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:46:22.514710 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.514589    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e923128-5de4-47ad-bb8c-665680a6cbd4" (UID: "8e923128-5de4-47ad-bb8c-665680a6cbd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 19:46:22.514710 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.514608    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e923128-5de4-47ad-bb8c-665680a6cbd4" (UID: "8e923128-5de4-47ad-bb8c-665680a6cbd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:46:22.514824 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.514801    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e923128-5de4-47ad-bb8c-665680a6cbd4" (UID: "8e923128-5de4-47ad-bb8c-665680a6cbd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:22.613818 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.613778    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-workload-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:22.613818 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.613810    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-data\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:22.613818 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.613821    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-credential-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:22.614071 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.613834    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-envoy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:22.614071 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.613845    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-podinfo\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:22.614071 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.613855    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp225\" (UniqueName: \"kubernetes.io/projected/8e923128-5de4-47ad-bb8c-665680a6cbd4-kube-api-access-xp225\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:22.614071 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.613864    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e923128-5de4-47ad-bb8c-665680a6cbd4-istio-token\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:22.614071 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.613873    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e923128-5de4-47ad-bb8c-665680a6cbd4-istiod-ca-cert\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:22.719722 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.719689    2571 generic.go:356] "Generic (PLEG): container finished" podID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerID="f8456d43249f71280e4fea4c399f45d73a7a91e6bcde99f23144f65bc928378b" exitCode=0
Sep 06 19:46:22.719896 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.719736    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"
Sep 06 19:46:22.719896 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.719768    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" event={"ID":"8e923128-5de4-47ad-bb8c-665680a6cbd4","Type":"ContainerDied","Data":"f8456d43249f71280e4fea4c399f45d73a7a91e6bcde99f23144f65bc928378b"}
Sep 06 19:46:22.719896 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.719857    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9" event={"ID":"8e923128-5de4-47ad-bb8c-665680a6cbd4","Type":"ContainerDied","Data":"5ab01b574689af17fe7dac1c6016459e0a498b49fbea35526549e39ac68680fc"}
Sep 06 19:46:22.719896 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.719882    2571 scope.go:117] "RemoveContainer" containerID="f8456d43249f71280e4fea4c399f45d73a7a91e6bcde99f23144f65bc928378b"
Sep 06 19:46:22.729255 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.729236    2571 scope.go:117] "RemoveContainer" containerID="12c581a817fae33aaccf7b8d08984d7787905e88eb4a6c8bd9e05269169f737e"
Sep 06 19:46:22.737489 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.737470    2571 scope.go:117] "RemoveContainer" containerID="f8456d43249f71280e4fea4c399f45d73a7a91e6bcde99f23144f65bc928378b"
Sep 06 19:46:22.737802 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:46:22.737781    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8456d43249f71280e4fea4c399f45d73a7a91e6bcde99f23144f65bc928378b\": container with ID starting with f8456d43249f71280e4fea4c399f45d73a7a91e6bcde99f23144f65bc928378b not found: ID does not exist" containerID="f8456d43249f71280e4fea4c399f45d73a7a91e6bcde99f23144f65bc928378b"
Sep 06 19:46:22.737859 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.737811    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8456d43249f71280e4fea4c399f45d73a7a91e6bcde99f23144f65bc928378b"} err="failed to get container status \"f8456d43249f71280e4fea4c399f45d73a7a91e6bcde99f23144f65bc928378b\": rpc error: code = NotFound desc = could not find container \"f8456d43249f71280e4fea4c399f45d73a7a91e6bcde99f23144f65bc928378b\": container with ID starting with f8456d43249f71280e4fea4c399f45d73a7a91e6bcde99f23144f65bc928378b not found: ID does not exist"
Sep 06 19:46:22.737859 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.737832    2571 scope.go:117] "RemoveContainer" containerID="12c581a817fae33aaccf7b8d08984d7787905e88eb4a6c8bd9e05269169f737e"
Sep 06 19:46:22.738104 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:46:22.738085    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c581a817fae33aaccf7b8d08984d7787905e88eb4a6c8bd9e05269169f737e\": container with ID starting with 12c581a817fae33aaccf7b8d08984d7787905e88eb4a6c8bd9e05269169f737e not found: ID does not exist" containerID="12c581a817fae33aaccf7b8d08984d7787905e88eb4a6c8bd9e05269169f737e"
Sep 06 19:46:22.738203 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.738104    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c581a817fae33aaccf7b8d08984d7787905e88eb4a6c8bd9e05269169f737e"} err="failed to get container status \"12c581a817fae33aaccf7b8d08984d7787905e88eb4a6c8bd9e05269169f737e\": rpc error: code = NotFound desc = could not find container \"12c581a817fae33aaccf7b8d08984d7787905e88eb4a6c8bd9e05269169f737e\": container with ID starting with 12c581a817fae33aaccf7b8d08984d7787905e88eb4a6c8bd9e05269169f737e not found: ID does not exist"
Sep 06 19:46:22.741134 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.741110    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"]
Sep 06 19:46:22.744606 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.744584    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-eb3b3780-openshift-default-6948cb88bd-54zm9"]
Sep 06 19:46:22.918150 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:22.918119    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" path="/var/lib/kubelet/pods/8e923128-5de4-47ad-bb8c-665680a6cbd4/volumes"
Sep 06 19:46:26.040756 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:26.040715    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:46:26.452249 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:26.452212    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"]
Sep 06 19:46:26.452493 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:26.452463    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq" podUID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" containerName="istio-proxy" containerID="cri-o://23c5c72b2fde08ed7b754bfa3ee7b4f6b8ba599b497c99dd5afd1c19480f2019" gracePeriod=30
Sep 06 19:46:31.707640 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.707609    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:46:31.767380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.767279    2571 generic.go:356] "Generic (PLEG): container finished" podID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" containerID="23c5c72b2fde08ed7b754bfa3ee7b4f6b8ba599b497c99dd5afd1c19480f2019" exitCode=0
Sep 06 19:46:31.767380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.767325    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"
Sep 06 19:46:31.767380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.767361    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq" event={"ID":"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d","Type":"ContainerDied","Data":"23c5c72b2fde08ed7b754bfa3ee7b4f6b8ba599b497c99dd5afd1c19480f2019"}
Sep 06 19:46:31.767605 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.767415    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq" event={"ID":"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d","Type":"ContainerDied","Data":"1ad233ed379a5735df7edee301c2bbac359a1d8acea378abea16b8a18a0c8ad6"}
Sep 06 19:46:31.767605 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.767439    2571 scope.go:117] "RemoveContainer" containerID="23c5c72b2fde08ed7b754bfa3ee7b4f6b8ba599b497c99dd5afd1c19480f2019"
Sep 06 19:46:31.776214 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.776195    2571 scope.go:117] "RemoveContainer" containerID="1ddd37391e7f4ec28a6ad89893f7e1a7fdeffbb7d75ba2dcb9ca887837a7e484"
Sep 06 19:46:31.784288 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.784265    2571 scope.go:117] "RemoveContainer" containerID="23c5c72b2fde08ed7b754bfa3ee7b4f6b8ba599b497c99dd5afd1c19480f2019"
Sep 06 19:46:31.784578 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:46:31.784557    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c5c72b2fde08ed7b754bfa3ee7b4f6b8ba599b497c99dd5afd1c19480f2019\": container with ID starting with 23c5c72b2fde08ed7b754bfa3ee7b4f6b8ba599b497c99dd5afd1c19480f2019 not found: ID does not exist" containerID="23c5c72b2fde08ed7b754bfa3ee7b4f6b8ba599b497c99dd5afd1c19480f2019"
Sep 06 19:46:31.784674 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.784590    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c5c72b2fde08ed7b754bfa3ee7b4f6b8ba599b497c99dd5afd1c19480f2019"} err="failed to get container status \"23c5c72b2fde08ed7b754bfa3ee7b4f6b8ba599b497c99dd5afd1c19480f2019\": rpc error: code = NotFound desc = could not find container \"23c5c72b2fde08ed7b754bfa3ee7b4f6b8ba599b497c99dd5afd1c19480f2019\": container with ID starting with 23c5c72b2fde08ed7b754bfa3ee7b4f6b8ba599b497c99dd5afd1c19480f2019 not found: ID does not exist"
Sep 06 19:46:31.784674 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.784609    2571 scope.go:117] "RemoveContainer" containerID="1ddd37391e7f4ec28a6ad89893f7e1a7fdeffbb7d75ba2dcb9ca887837a7e484"
Sep 06 19:46:31.784888 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:46:31.784870    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ddd37391e7f4ec28a6ad89893f7e1a7fdeffbb7d75ba2dcb9ca887837a7e484\": container with ID starting with 1ddd37391e7f4ec28a6ad89893f7e1a7fdeffbb7d75ba2dcb9ca887837a7e484 not found: ID does not exist" containerID="1ddd37391e7f4ec28a6ad89893f7e1a7fdeffbb7d75ba2dcb9ca887837a7e484"
Sep 06 19:46:31.784937 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.784897    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ddd37391e7f4ec28a6ad89893f7e1a7fdeffbb7d75ba2dcb9ca887837a7e484"} err="failed to get container status \"1ddd37391e7f4ec28a6ad89893f7e1a7fdeffbb7d75ba2dcb9ca887837a7e484\": rpc error: code = NotFound desc = could not find container \"1ddd37391e7f4ec28a6ad89893f7e1a7fdeffbb7d75ba2dcb9ca887837a7e484\": container with ID starting with 1ddd37391e7f4ec28a6ad89893f7e1a7fdeffbb7d75ba2dcb9ca887837a7e484 not found: ID does not exist"
Sep 06 19:46:31.804920 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.804878    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-data\") pod \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") "
Sep 06 19:46:31.805117 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.805018    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-token\") pod \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") "
Sep 06 19:46:31.805117 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.805040    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-envoy\") pod \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") "
Sep 06 19:46:31.805117 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.805067    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-credential-socket\") pod \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") "
Sep 06 19:46:31.805117 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.805098    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-podinfo\") pod \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") "
Sep 06 19:46:31.805331 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.805132    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istiod-ca-cert\") pod \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") "
Sep 06 19:46:31.805331 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.805156    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh764\" (UniqueName: \"kubernetes.io/projected/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-kube-api-access-mh764\") pod \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") "
Sep 06 19:46:31.805331 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.805184    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-workload-certs\") pod \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") "
Sep 06 19:46:31.805331 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.805186    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" (UID: "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:31.805331 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.805228    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-workload-socket\") pod \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\" (UID: \"1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d\") "
Sep 06 19:46:31.805585 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.805385    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" (UID: "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:31.805585 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.805551    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-data\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:31.805585 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.805558    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" (UID: "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:31.805585 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.805574    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-credential-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:31.805585 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.805570    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" (UID: "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:31.805876 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.805849    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" (UID: "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:46:31.807326 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.807302    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-kube-api-access-mh764" (OuterVolumeSpecName: "kube-api-access-mh764") pod "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" (UID: "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d"). InnerVolumeSpecName "kube-api-access-mh764". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:46:31.807435 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.807417    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" (UID: "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 19:46:31.807670 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.807654    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" (UID: "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:46:31.807712 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.807680    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" (UID: "1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:46:31.906670 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.906613    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-token\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:31.906670 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.906660    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-envoy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:31.906670 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.906670    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istio-podinfo\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:31.906670 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.906678    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-istiod-ca-cert\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:31.906670 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.906688    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh764\" (UniqueName: \"kubernetes.io/projected/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-kube-api-access-mh764\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:31.907024 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.906698    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-workload-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:31.907024 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:31.906706    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d-workload-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:46:32.095838 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:32.095785    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"]
Sep 06 19:46:32.098055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:32.098023    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d5bb2d-openshift-default-68d4cc5d54-9ldlq"]
Sep 06 19:46:32.917772 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:32.917727    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" path="/var/lib/kubelet/pods/1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d/volumes"
Sep 06 19:46:35.153897 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:35.153855    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:46:44.836551 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:44.836462    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:46:56.162642 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.162599    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"]
Sep 06 19:46:56.163188 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.163015    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerName="istio-proxy"
Sep 06 19:46:56.163188 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.163029    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerName="istio-proxy"
Sep 06 19:46:56.163188 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.163037    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2" containerName="revoke-keys"
Sep 06 19:46:56.163188 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.163042    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2" containerName="revoke-keys"
Sep 06 19:46:56.163188 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.163059    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerName="istio-proxy"
Sep 06 19:46:56.163188 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.163067    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerName="istio-proxy"
Sep 06 19:46:56.163188 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.163077    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" containerName="istio-proxy"
Sep 06 19:46:56.163188 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.163082    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" containerName="istio-proxy"
Sep 06 19:46:56.163188 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.163097    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" containerName="istio-proxy"
Sep 06 19:46:56.163188 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.163106    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" containerName="istio-proxy"
Sep 06 19:46:56.164073 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.163198    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7afaaee8-08a7-4af7-9cec-89fcd0eaf9c2" containerName="revoke-keys"
Sep 06 19:46:56.164073 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.163217    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" containerName="istio-proxy"
Sep 06 19:46:56.164073 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.163226    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerName="istio-proxy"
Sep 06 19:46:56.164073 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.163235    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a7ed1e6-50ff-4dcb-ba4c-f91af64f806d" containerName="istio-proxy"
Sep 06 19:46:56.164073 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.163241    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e923128-5de4-47ad-bb8c-665680a6cbd4" containerName="istio-proxy"
Sep 06 19:46:56.166601 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.166577    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.168952 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.168927    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-712616-openshift-default-dockercfg-mc6w9\""
Sep 06 19:46:56.177285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.177256    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"]
Sep 06 19:46:56.241318 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.241279    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-token\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.241490 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.241326    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-workload-certs\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.241490 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.241420    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-podinfo\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.241490 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.241457    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-credential-socket\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.241490 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.241483    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066db7cd-7ba1-49d0-8539-2f6cde582dae-istiod-ca-cert\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.241630 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.241542    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn48h\" (UniqueName: \"kubernetes.io/projected/066db7cd-7ba1-49d0-8539-2f6cde582dae-kube-api-access-cn48h\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.241630 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.241593    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-envoy\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.241630 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.241614    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-data\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.241724 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.241683    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-workload-socket\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.342734 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.342681    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-envoy\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.342734 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.342738    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-data\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.343032 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.342771    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-workload-socket\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.343032 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.342800    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-token\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.343032 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.342822    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-workload-certs\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.343032 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.342858    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-podinfo\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.343032 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.342886    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-credential-socket\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.343032 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.342912    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066db7cd-7ba1-49d0-8539-2f6cde582dae-istiod-ca-cert\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.343032 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.343001    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn48h\" (UniqueName: \"kubernetes.io/projected/066db7cd-7ba1-49d0-8539-2f6cde582dae-kube-api-access-cn48h\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.343384 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.343242    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-data\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.343384 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.343296    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-workload-certs\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.343484 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.343456    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-workload-socket\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.343563 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.343548    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-credential-socket\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.343785 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.343763    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066db7cd-7ba1-49d0-8539-2f6cde582dae-istiod-ca-cert\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.345210 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.345189    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-envoy\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.345395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.345375    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-podinfo\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.351003 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.350979    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-token\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.351131 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.351114    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn48h\" (UniqueName: \"kubernetes.io/projected/066db7cd-7ba1-49d0-8539-2f6cde582dae-kube-api-access-cn48h\") pod \"e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") " pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.478203 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.478117    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:56.751806 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.751780    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"]
Sep 06 19:46:56.753870 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:46:56.753843    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066db7cd_7ba1_49d0_8539_2f6cde582dae.slice/crio-1eef30594bdabdca148795a85c3ff739d036e420e9dc02ea064078d8bc3ed2a1 WatchSource:0}: Error finding container 1eef30594bdabdca148795a85c3ff739d036e420e9dc02ea064078d8bc3ed2a1: Status 404 returned error can't find the container with id 1eef30594bdabdca148795a85c3ff739d036e420e9dc02ea064078d8bc3ed2a1
Sep 06 19:46:56.756430 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.756395    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:46:56.756529 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.756481    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:46:56.756597 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.756526    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:46:56.902603 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.902564    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6" event={"ID":"066db7cd-7ba1-49d0-8539-2f6cde582dae","Type":"ContainerStarted","Data":"222c93b303556e4b28665502add137dde1a7524559db37aa80d3fa12ffca58f5"}
Sep 06 19:46:56.902796 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.902629    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6" event={"ID":"066db7cd-7ba1-49d0-8539-2f6cde582dae","Type":"ContainerStarted","Data":"1eef30594bdabdca148795a85c3ff739d036e420e9dc02ea064078d8bc3ed2a1"}
Sep 06 19:46:56.918838 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:56.918779    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6" podStartSLOduration=0.918754985 podStartE2EDuration="918.754985ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:46:56.918555263 +0000 UTC m=+1180.488853669" watchObservedRunningTime="2026-09-06 19:46:56.918754985 +0000 UTC m=+1180.489053329"
Sep 06 19:46:57.478938 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:57.478898    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:57.484144 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:57.484116    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:57.907825 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:57.907794    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:57.908997 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:57.908969    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:46:59.291749 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.291716    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"]
Sep 06 19:46:59.295376 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.295350    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.297540 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.297512    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6ddeb5-openshift-default-dockercfg-zxvjz\""
Sep 06 19:46:59.306630 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.306597    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"]
Sep 06 19:46:59.368712 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.368670    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw5lh\" (UniqueName: \"kubernetes.io/projected/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-kube-api-access-fw5lh\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.368904 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.368746    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-podinfo\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.368904 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.368795    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istiod-ca-cert\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.368904 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.368827    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-token\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.368904 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.368847    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-data\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.368904 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.368867    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-workload-socket\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.369136 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.368989    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-workload-certs\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.369136 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.369040    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-envoy\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.369136 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.369081    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-credential-socket\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.470333 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.470291    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-podinfo\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.470333 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.470334    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istiod-ca-cert\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.470608 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.470353    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-token\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.470608 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.470370    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-data\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.470608 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.470389    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-workload-socket\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.470608 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.470449    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-workload-certs\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.470608 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.470497    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-envoy\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.470608 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.470543    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-credential-socket\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.470608 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.470604    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw5lh\" (UniqueName: \"kubernetes.io/projected/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-kube-api-access-fw5lh\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.471047 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.470891    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-data\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.471047 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.470950    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-credential-socket\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.471047 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.471023    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-workload-certs\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.471216 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.471174    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istiod-ca-cert\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.471261 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.471220    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-workload-socket\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.473109 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.473089    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-podinfo\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.473268 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.473178    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-envoy\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.477313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.477290    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-token\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.477496 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.477477    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw5lh\" (UniqueName: \"kubernetes.io/projected/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-kube-api-access-fw5lh\") pod \"e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") " pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.608796 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.608751    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:46:59.907411 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.907383    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"]
Sep 06 19:46:59.909194 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:46:59.909164    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b62b4d1_ecd0_47b2_b79b_01d1b4bef961.slice/crio-a528b87250a3082899906bbef34079ff56fd0a55ecf1b2338e2b28ec53b17824 WatchSource:0}: Error finding container a528b87250a3082899906bbef34079ff56fd0a55ecf1b2338e2b28ec53b17824: Status 404 returned error can't find the container with id a528b87250a3082899906bbef34079ff56fd0a55ecf1b2338e2b28ec53b17824
Sep 06 19:46:59.912457 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.912418    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:46:59.912570 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.912497    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:46:59.912570 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.912550    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:46:59.921517 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:46:59.921484    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq" event={"ID":"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961","Type":"ContainerStarted","Data":"a528b87250a3082899906bbef34079ff56fd0a55ecf1b2338e2b28ec53b17824"}
Sep 06 19:47:00.927449 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:00.927411    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq" event={"ID":"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961","Type":"ContainerStarted","Data":"99c9f2af719ef054a1b3aa6e1cb06aa92cc97c2f48dfcfebb6a3ff516b024977"}
Sep 06 19:47:00.945122 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:00.945070    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq" podStartSLOduration=1.945054037 podStartE2EDuration="1.945054037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:47:00.944121207 +0000 UTC m=+1184.514419548" watchObservedRunningTime="2026-09-06 19:47:00.945054037 +0000 UTC m=+1184.515352383"
Sep 06 19:47:01.608975 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:01.608916    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:47:01.614272 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:01.614239    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:47:01.932681 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:01.932589    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:47:01.933756 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:01.933739    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:47:03.940627 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:03.940593    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"]
Sep 06 19:47:03.944743 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:03.944716    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:03.946746 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:03.946720    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-6ddeb5-serving-cert\""
Sep 06 19:47:03.948359 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:03.948326    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 06 19:47:03.948482 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:03.948445    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bfdfp\""
Sep 06 19:47:03.948562 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:03.948381    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 06 19:47:03.948624 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:03.948331    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 19:47:03.965923 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:03.965878    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"]
Sep 06 19:47:04.118315 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.118277    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f498a7d9-d881-4050-8d12-86684816e5fb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:04.118483 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.118357    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f498a7d9-d881-4050-8d12-86684816e5fb-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:04.118526 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.118481    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f498a7d9-d881-4050-8d12-86684816e5fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:04.118526 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.118513    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzcq\" (UniqueName: \"kubernetes.io/projected/f498a7d9-d881-4050-8d12-86684816e5fb-kube-api-access-dvzcq\") pod \"maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:04.219502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.219391    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f498a7d9-d881-4050-8d12-86684816e5fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:04.219502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.219437    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzcq\" (UniqueName: \"kubernetes.io/projected/f498a7d9-d881-4050-8d12-86684816e5fb-kube-api-access-dvzcq\") pod \"maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:04.219502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.219491    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f498a7d9-d881-4050-8d12-86684816e5fb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:04.219768 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.219540    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f498a7d9-d881-4050-8d12-86684816e5fb-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:04.220264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.220233    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f498a7d9-d881-4050-8d12-86684816e5fb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:04.222101 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.222073    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f498a7d9-d881-4050-8d12-86684816e5fb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:04.222205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.222075    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f498a7d9-d881-4050-8d12-86684816e5fb-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:04.226940 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.226914    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzcq\" (UniqueName: \"kubernetes.io/projected/f498a7d9-d881-4050-8d12-86684816e5fb-kube-api-access-dvzcq\") pod \"maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:04.268803 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.268761    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:04.543036 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.543011    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"]
Sep 06 19:47:04.544842 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:47:04.544808    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf498a7d9_d881_4050_8d12_86684816e5fb.slice/crio-8dbe1d71dfee4d34365350b50f14e5d84f810febe434537e516e9f734e20eca5 WatchSource:0}: Error finding container 8dbe1d71dfee4d34365350b50f14e5d84f810febe434537e516e9f734e20eca5: Status 404 returned error can't find the container with id 8dbe1d71dfee4d34365350b50f14e5d84f810febe434537e516e9f734e20eca5
Sep 06 19:47:04.546433 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.546410    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 19:47:04.950546 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:04.950512    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp" event={"ID":"f498a7d9-d881-4050-8d12-86684816e5fb","Type":"ContainerStarted","Data":"8dbe1d71dfee4d34365350b50f14e5d84f810febe434537e516e9f734e20eca5"}
Sep 06 19:47:06.965686 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:06.965640    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp" event={"ID":"f498a7d9-d881-4050-8d12-86684816e5fb","Type":"ContainerStarted","Data":"b8d1c78473cf1ea420b97ef75e954fa4da52754861a5f2113c2c520f3effc27c"}
Sep 06 19:47:06.966162 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:06.965707    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:06.983065 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:06.983008    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp" podStartSLOduration=1.728111482 podStartE2EDuration="3.982992376s" totalImagesPullingTime="2.254880894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:47:06.981569333 +0000 UTC m=+1190.551867681" watchObservedRunningTime="2026-09-06 19:47:06.982992376 +0000 UTC m=+1190.553290722"
Sep 06 19:47:07.647687 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:07.647651    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:47:12.982528 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:12.982488    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:47:16.956106 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:16.956061    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 19:47:16.956971 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:16.956945    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 19:47:17.975120 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:17.975058    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 06 19:47:17.975606 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:17.975147    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq" podUID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 06 19:47:18.042804 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:18.042771    2571 generic.go:356] "Generic (PLEG): container finished" podID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" containerID="99c9f2af719ef054a1b3aa6e1cb06aa92cc97c2f48dfcfebb6a3ff516b024977" exitCode=0
Sep 06 19:47:18.043975 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:18.042818    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq" event={"ID":"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961","Type":"ContainerDied","Data":"99c9f2af719ef054a1b3aa6e1cb06aa92cc97c2f48dfcfebb6a3ff516b024977"}
Sep 06 19:47:18.043975 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:18.043251    2571 scope.go:117] "RemoveContainer" containerID="99c9f2af719ef054a1b3aa6e1cb06aa92cc97c2f48dfcfebb6a3ff516b024977"
Sep 06 19:47:18.044479 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:18.044381    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:47:18.044479 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:18.044448    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:47:18.044594 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:18.044479    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:47:19.057264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:19.057219    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq" event={"ID":"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961","Type":"ContainerStarted","Data":"2b62ef1f6e7e637e9c69319d88a090e47253a4824d7be14ef4e99652cfcade1f"}
Sep 06 19:47:19.060056 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:19.060028    2571 generic.go:356] "Generic (PLEG): container finished" podID="066db7cd-7ba1-49d0-8539-2f6cde582dae" containerID="222c93b303556e4b28665502add137dde1a7524559db37aa80d3fa12ffca58f5" exitCode=0
Sep 06 19:47:19.060206 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:19.060067    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6" event={"ID":"066db7cd-7ba1-49d0-8539-2f6cde582dae","Type":"ContainerDied","Data":"222c93b303556e4b28665502add137dde1a7524559db37aa80d3fa12ffca58f5"}
Sep 06 19:47:19.060416 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:19.060402    2571 scope.go:117] "RemoveContainer" containerID="222c93b303556e4b28665502add137dde1a7524559db37aa80d3fa12ffca58f5"
Sep 06 19:47:19.061663 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:19.061634    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:47:19.061754 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:19.061701    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:47:19.061754 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:19.061736    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:47:19.609510 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:19.609467    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:47:19.609845 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:19.609820    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 06 19:47:19.609910 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:19.609881    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq" podUID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 06 19:47:20.077264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:20.077221    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6" event={"ID":"066db7cd-7ba1-49d0-8539-2f6cde582dae","Type":"ContainerStarted","Data":"1afc58af76fe5a3422a012941b51edfd8f98177f658d7363b75ac9c200c2bae1"}
Sep 06 19:47:20.479248 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:20.479144    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:47:20.479599 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:20.479572    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 06 19:47:20.479762 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:20.479736    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6" podUID="066db7cd-7ba1-49d0-8539-2f6cde582dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 06 19:47:20.609908 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:20.609862    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 06 19:47:20.610143 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:20.609923    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq" podUID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 06 19:47:21.482423 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:21.482378    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 06 19:47:21.483040 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:21.483004    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6" podUID="066db7cd-7ba1-49d0-8539-2f6cde582dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 06 19:47:21.609531 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:21.609491    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 06 19:47:21.609702 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:21.609545    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq" podUID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 06 19:47:22.479664 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:22.479615    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 06 19:47:22.479897 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:22.479697    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6" podUID="066db7cd-7ba1-49d0-8539-2f6cde582dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 06 19:47:22.609556 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:22.609515    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 06 19:47:22.610013 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:22.609603    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq" podUID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 06 19:47:23.479627 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:23.479587    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 06 19:47:23.480378 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:23.479944    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6" podUID="066db7cd-7ba1-49d0-8539-2f6cde582dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 06 19:47:23.609571 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:23.609533    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 06 19:47:23.609996 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:23.609596    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq" podUID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 06 19:47:24.479144 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:24.479104    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 06 19:47:24.479327 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:24.479165    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6" podUID="066db7cd-7ba1-49d0-8539-2f6cde582dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 06 19:47:24.613612 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:24.613581    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:47:24.614005 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:24.613990    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:47:24.614774 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:24.614752    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:47:25.478757 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:25.478716    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 06 19:47:25.478983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:25.478784    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6" podUID="066db7cd-7ba1-49d0-8539-2f6cde582dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 06 19:47:26.483328 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:26.483291    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:47:26.483704 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:26.483658    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:47:26.484347 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:26.484331    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:47:28.335911 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:28.335874    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7636-140191469527520"]
Sep 06 19:47:28.343195 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:28.343166    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469527520"
Sep 06 19:47:28.344535 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:28.344502    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-140191469527520"]
Sep 06 19:47:28.345539 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:28.345518    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kn2d6\""
Sep 06 19:47:28.464147 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:28.464105    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb2v2\" (UniqueName: \"kubernetes.io/projected/c1b0951c-d0ec-49e1-8c03-0a4b8e349b57-kube-api-access-rb2v2\") pod \"test-curl-7636-140191469527520\" (UID: \"c1b0951c-d0ec-49e1-8c03-0a4b8e349b57\") " pod="opendatahub/test-curl-7636-140191469527520"
Sep 06 19:47:28.565249 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:28.565213    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb2v2\" (UniqueName: \"kubernetes.io/projected/c1b0951c-d0ec-49e1-8c03-0a4b8e349b57-kube-api-access-rb2v2\") pod \"test-curl-7636-140191469527520\" (UID: \"c1b0951c-d0ec-49e1-8c03-0a4b8e349b57\") " pod="opendatahub/test-curl-7636-140191469527520"
Sep 06 19:47:28.572512 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:28.572484    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb2v2\" (UniqueName: \"kubernetes.io/projected/c1b0951c-d0ec-49e1-8c03-0a4b8e349b57-kube-api-access-rb2v2\") pod \"test-curl-7636-140191469527520\" (UID: \"c1b0951c-d0ec-49e1-8c03-0a4b8e349b57\") " pod="opendatahub/test-curl-7636-140191469527520"
Sep 06 19:47:28.654986 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:28.654887    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469527520"
Sep 06 19:47:28.921732 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:28.921706    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-140191469527520"]
Sep 06 19:47:28.923592 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:47:28.923556    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1b0951c_d0ec_49e1_8c03_0a4b8e349b57.slice/crio-3b20f6e1699477092e2f86aa7657c8c13fcf743f1861d67cb5eec12478d38350 WatchSource:0}: Error finding container 3b20f6e1699477092e2f86aa7657c8c13fcf743f1861d67cb5eec12478d38350: Status 404 returned error can't find the container with id 3b20f6e1699477092e2f86aa7657c8c13fcf743f1861d67cb5eec12478d38350
Sep 06 19:47:29.131471 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:29.131439    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-140191469527520" event={"ID":"c1b0951c-d0ec-49e1-8c03-0a4b8e349b57","Type":"ContainerStarted","Data":"3b20f6e1699477092e2f86aa7657c8c13fcf743f1861d67cb5eec12478d38350"}
Sep 06 19:47:30.138294 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:30.138258    2571 generic.go:356] "Generic (PLEG): container finished" podID="c1b0951c-d0ec-49e1-8c03-0a4b8e349b57" containerID="8e4dcb21bb5de7b00bb5b0bba98d4910de49c744196d3d266ad543453dd9294e" exitCode=0
Sep 06 19:47:30.138750 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:30.138328    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-140191469527520" event={"ID":"c1b0951c-d0ec-49e1-8c03-0a4b8e349b57","Type":"ContainerDied","Data":"8e4dcb21bb5de7b00bb5b0bba98d4910de49c744196d3d266ad543453dd9294e"}
Sep 06 19:47:30.939265 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:30.939229    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"]
Sep 06 19:47:30.942864 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:30.942847    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:30.946091 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:30.946063    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b571c4\"/\"auth-test-model-b571c4-kserve-self-signed-certs\""
Sep 06 19:47:30.946235 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:30.946094    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b571c4\"/\"kube-root-ca.crt\""
Sep 06 19:47:30.946235 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:30.946111    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b571c4\"/\"default-dockercfg-7dl84\""
Sep 06 19:47:30.946235 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:30.946064    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b571c4\"/\"openshift-service-ca.crt\""
Sep 06 19:47:30.950831 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:30.950585    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"]
Sep 06 19:47:30.990917 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:30.990870    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-tmp-dir\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:30.991139 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:30.991000    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-home\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:30.991139 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:30.991033    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-dshm\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:30.991139 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:30.991120    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8bv5\" (UniqueName: \"kubernetes.io/projected/c6e8086c-3282-464e-bc93-937a3aa06ab0-kube-api-access-t8bv5\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:30.991303 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:30.991150    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e8086c-3282-464e-bc93-937a3aa06ab0-tls-certs\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:30.991303 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:30.991248    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-model-cache\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:31.092571 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.092528    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-home\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:31.092571 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.092572    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-dshm\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:31.092845 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.092611    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8bv5\" (UniqueName: \"kubernetes.io/projected/c6e8086c-3282-464e-bc93-937a3aa06ab0-kube-api-access-t8bv5\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:31.092845 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.092636    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e8086c-3282-464e-bc93-937a3aa06ab0-tls-certs\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:31.092845 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.092668    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-model-cache\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:31.092845 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.092725    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-tmp-dir\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:31.093105 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.093056    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-home\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:31.093174 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.093153    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-tmp-dir\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:31.093210 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.093169    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-model-cache\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:31.094975 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.094933    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-dshm\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:31.095221 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.095205    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e8086c-3282-464e-bc93-937a3aa06ab0-tls-certs\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:31.100300 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.100282    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8bv5\" (UniqueName: \"kubernetes.io/projected/c6e8086c-3282-464e-bc93-937a3aa06ab0-kube-api-access-t8bv5\") pod \"auth-test-model-b571c4-kserve-66b78f66c5-r6gr7\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:31.254257 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.254221    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:31.290168 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.290139    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469527520"
Sep 06 19:47:31.320835 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.320810    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7636-140191469527520_c1b0951c-d0ec-49e1-8c03-0a4b8e349b57/test-curl-7636-140191469527520/0.log"
Sep 06 19:47:31.339924 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.339889    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7636-140191469527520"]
Sep 06 19:47:31.343723 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.343698    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7636-140191469527520"]
Sep 06 19:47:31.395191 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.395150    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb2v2\" (UniqueName: \"kubernetes.io/projected/c1b0951c-d0ec-49e1-8c03-0a4b8e349b57-kube-api-access-rb2v2\") pod \"c1b0951c-d0ec-49e1-8c03-0a4b8e349b57\" (UID: \"c1b0951c-d0ec-49e1-8c03-0a4b8e349b57\") "
Sep 06 19:47:31.397455 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.397417    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b0951c-d0ec-49e1-8c03-0a4b8e349b57-kube-api-access-rb2v2" (OuterVolumeSpecName: "kube-api-access-rb2v2") pod "c1b0951c-d0ec-49e1-8c03-0a4b8e349b57" (UID: "c1b0951c-d0ec-49e1-8c03-0a4b8e349b57"). InnerVolumeSpecName "kube-api-access-rb2v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:47:31.459295 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.459213    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7636-140191469527520"]
Sep 06 19:47:31.459619 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.459606    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b0951c-d0ec-49e1-8c03-0a4b8e349b57" containerName="test-curl-7636-140191469527520"
Sep 06 19:47:31.459671 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.459621    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b0951c-d0ec-49e1-8c03-0a4b8e349b57" containerName="test-curl-7636-140191469527520"
Sep 06 19:47:31.459707 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.459683    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b0951c-d0ec-49e1-8c03-0a4b8e349b57" containerName="test-curl-7636-140191469527520"
Sep 06 19:47:31.462661 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.462644    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469527520"
Sep 06 19:47:31.469893 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.469704    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-140191469527520"]
Sep 06 19:47:31.495974 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.495915    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb2v2\" (UniqueName: \"kubernetes.io/projected/c1b0951c-d0ec-49e1-8c03-0a4b8e349b57-kube-api-access-rb2v2\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:47:31.527777 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.527747    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"]
Sep 06 19:47:31.529429 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:47:31.529397    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6e8086c_3282_464e_bc93_937a3aa06ab0.slice/crio-6663f2ac24bace297c75680a95bc1c710b757ce1db3876cdc8696adc6514d7f0 WatchSource:0}: Error finding container 6663f2ac24bace297c75680a95bc1c710b757ce1db3876cdc8696adc6514d7f0: Status 404 returned error can't find the container with id 6663f2ac24bace297c75680a95bc1c710b757ce1db3876cdc8696adc6514d7f0
Sep 06 19:47:31.596764 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.596730    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4cxz\" (UniqueName: \"kubernetes.io/projected/fc7497e1-0d3d-49c9-8e47-f669687b4b06-kube-api-access-t4cxz\") pod \"test-curl-7636-140191469527520\" (UID: \"fc7497e1-0d3d-49c9-8e47-f669687b4b06\") " pod="opendatahub/test-curl-7636-140191469527520"
Sep 06 19:47:31.698101 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.698049    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4cxz\" (UniqueName: \"kubernetes.io/projected/fc7497e1-0d3d-49c9-8e47-f669687b4b06-kube-api-access-t4cxz\") pod \"test-curl-7636-140191469527520\" (UID: \"fc7497e1-0d3d-49c9-8e47-f669687b4b06\") " pod="opendatahub/test-curl-7636-140191469527520"
Sep 06 19:47:31.705645 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.705608    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4cxz\" (UniqueName: \"kubernetes.io/projected/fc7497e1-0d3d-49c9-8e47-f669687b4b06-kube-api-access-t4cxz\") pod \"test-curl-7636-140191469527520\" (UID: \"fc7497e1-0d3d-49c9-8e47-f669687b4b06\") " pod="opendatahub/test-curl-7636-140191469527520"
Sep 06 19:47:31.777631 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:31.777531    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469527520"
Sep 06 19:47:32.046192 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:32.046165    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-140191469527520"]
Sep 06 19:47:32.048520 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:47:32.048488    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc7497e1_0d3d_49c9_8e47_f669687b4b06.slice/crio-af9da0f2fc80b32c1b690cd3641e3e36cc9ffd03f9aaf5528867123742dd4263 WatchSource:0}: Error finding container af9da0f2fc80b32c1b690cd3641e3e36cc9ffd03f9aaf5528867123742dd4263: Status 404 returned error can't find the container with id af9da0f2fc80b32c1b690cd3641e3e36cc9ffd03f9aaf5528867123742dd4263
Sep 06 19:47:32.150509 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:32.150473    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7" event={"ID":"c6e8086c-3282-464e-bc93-937a3aa06ab0","Type":"ContainerStarted","Data":"519c5549445c2a205c7e4650be8d9c840349188972da4bd4ac23171bc843d8cd"}
Sep 06 19:47:32.150509 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:32.150515    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7" event={"ID":"c6e8086c-3282-464e-bc93-937a3aa06ab0","Type":"ContainerStarted","Data":"6663f2ac24bace297c75680a95bc1c710b757ce1db3876cdc8696adc6514d7f0"}
Sep 06 19:47:32.151821 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:32.151794    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-140191469527520" event={"ID":"fc7497e1-0d3d-49c9-8e47-f669687b4b06","Type":"ContainerStarted","Data":"af9da0f2fc80b32c1b690cd3641e3e36cc9ffd03f9aaf5528867123742dd4263"}
Sep 06 19:47:32.153026 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:32.153008    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b20f6e1699477092e2f86aa7657c8c13fcf743f1861d67cb5eec12478d38350"
Sep 06 19:47:32.153124 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:32.153062    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469527520"
Sep 06 19:47:32.167163 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:32.167116    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7" podStartSLOduration=2.167101238 podStartE2EDuration="2.167101238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:47:32.164985942 +0000 UTC m=+1215.735284278" watchObservedRunningTime="2026-09-06 19:47:32.167101238 +0000 UTC m=+1215.737399583"
Sep 06 19:47:32.167574 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:32.167549    2571 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7636-140191469527520" oldPodUID="c1b0951c-d0ec-49e1-8c03-0a4b8e349b57" podUID="fc7497e1-0d3d-49c9-8e47-f669687b4b06"
Sep 06 19:47:32.918666 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:32.918632    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1b0951c-d0ec-49e1-8c03-0a4b8e349b57" path="/var/lib/kubelet/pods/c1b0951c-d0ec-49e1-8c03-0a4b8e349b57/volumes"
Sep 06 19:47:33.159712 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:33.159678    2571 generic.go:356] "Generic (PLEG): container finished" podID="fc7497e1-0d3d-49c9-8e47-f669687b4b06" containerID="779fc2e80b6bc9b5792b1455f13c22744142f605f32bd42df0d380eb2707fc0b" exitCode=0
Sep 06 19:47:33.159906 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:33.159725    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-140191469527520" event={"ID":"fc7497e1-0d3d-49c9-8e47-f669687b4b06","Type":"ContainerDied","Data":"779fc2e80b6bc9b5792b1455f13c22744142f605f32bd42df0d380eb2707fc0b"}
Sep 06 19:47:34.040888 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.040848    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:47:34.309483 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.309455    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469527520"
Sep 06 19:47:34.346996 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.345434    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7636-140191469527520_fc7497e1-0d3d-49c9-8e47-f669687b4b06/test-curl-7636-140191469527520/0.log"
Sep 06 19:47:34.367967 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.367919    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7636-140191469527520"]
Sep 06 19:47:34.369697 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.369668    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7636-140191469527520"]
Sep 06 19:47:34.428231 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.428194    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4cxz\" (UniqueName: \"kubernetes.io/projected/fc7497e1-0d3d-49c9-8e47-f669687b4b06-kube-api-access-t4cxz\") pod \"fc7497e1-0d3d-49c9-8e47-f669687b4b06\" (UID: \"fc7497e1-0d3d-49c9-8e47-f669687b4b06\") "
Sep 06 19:47:34.430362 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.430325    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7497e1-0d3d-49c9-8e47-f669687b4b06-kube-api-access-t4cxz" (OuterVolumeSpecName: "kube-api-access-t4cxz") pod "fc7497e1-0d3d-49c9-8e47-f669687b4b06" (UID: "fc7497e1-0d3d-49c9-8e47-f669687b4b06"). InnerVolumeSpecName "kube-api-access-t4cxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:47:34.523608 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.523568    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7636-140191469528096"]
Sep 06 19:47:34.524224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.524191    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc7497e1-0d3d-49c9-8e47-f669687b4b06" containerName="test-curl-7636-140191469527520"
Sep 06 19:47:34.524224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.524221    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc7497e1-0d3d-49c9-8e47-f669687b4b06" containerName="test-curl-7636-140191469527520"
Sep 06 19:47:34.524386 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.524355    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc7497e1-0d3d-49c9-8e47-f669687b4b06" containerName="test-curl-7636-140191469527520"
Sep 06 19:47:34.527855 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.527833    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469528096"
Sep 06 19:47:34.529542 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.529512    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4cxz\" (UniqueName: \"kubernetes.io/projected/fc7497e1-0d3d-49c9-8e47-f669687b4b06-kube-api-access-t4cxz\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:47:34.539232 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.539204    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-140191469528096"]
Sep 06 19:47:34.630677 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.630639    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchj7\" (UniqueName: \"kubernetes.io/projected/f88e16ca-b62a-40a3-92b0-6102b5eeb91f-kube-api-access-hchj7\") pod \"test-curl-7636-140191469528096\" (UID: \"f88e16ca-b62a-40a3-92b0-6102b5eeb91f\") " pod="opendatahub/test-curl-7636-140191469528096"
Sep 06 19:47:34.732359 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.732321    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchj7\" (UniqueName: \"kubernetes.io/projected/f88e16ca-b62a-40a3-92b0-6102b5eeb91f-kube-api-access-hchj7\") pod \"test-curl-7636-140191469528096\" (UID: \"f88e16ca-b62a-40a3-92b0-6102b5eeb91f\") " pod="opendatahub/test-curl-7636-140191469528096"
Sep 06 19:47:34.739348 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.739321    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchj7\" (UniqueName: \"kubernetes.io/projected/f88e16ca-b62a-40a3-92b0-6102b5eeb91f-kube-api-access-hchj7\") pod \"test-curl-7636-140191469528096\" (UID: \"f88e16ca-b62a-40a3-92b0-6102b5eeb91f\") " pod="opendatahub/test-curl-7636-140191469528096"
Sep 06 19:47:34.838414 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.838375    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469528096"
Sep 06 19:47:34.920860 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:34.920819    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc7497e1-0d3d-49c9-8e47-f669687b4b06" path="/var/lib/kubelet/pods/fc7497e1-0d3d-49c9-8e47-f669687b4b06/volumes"
Sep 06 19:47:35.108381 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:35.108354    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-140191469528096"]
Sep 06 19:47:35.110231 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:47:35.110203    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf88e16ca_b62a_40a3_92b0_6102b5eeb91f.slice/crio-032530fceeba653336b9a8a66cf336c2d4deb95fb74dea66833d1e5960336e2c WatchSource:0}: Error finding container 032530fceeba653336b9a8a66cf336c2d4deb95fb74dea66833d1e5960336e2c: Status 404 returned error can't find the container with id 032530fceeba653336b9a8a66cf336c2d4deb95fb74dea66833d1e5960336e2c
Sep 06 19:47:35.171253 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:35.171171    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469527520"
Sep 06 19:47:35.171253 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:35.171202    2571 scope.go:117] "RemoveContainer" containerID="779fc2e80b6bc9b5792b1455f13c22744142f605f32bd42df0d380eb2707fc0b"
Sep 06 19:47:35.172639 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:35.172489    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-140191469528096" event={"ID":"f88e16ca-b62a-40a3-92b0-6102b5eeb91f","Type":"ContainerStarted","Data":"032530fceeba653336b9a8a66cf336c2d4deb95fb74dea66833d1e5960336e2c"}
Sep 06 19:47:36.181412 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:36.181381    2571 generic.go:356] "Generic (PLEG): container finished" podID="f88e16ca-b62a-40a3-92b0-6102b5eeb91f" containerID="a25afd14c697bdad643f7d9b68e9a92cb3d0b36560061004eec0ee73cf5e8588" exitCode=0
Sep 06 19:47:36.181786 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:36.181444    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-140191469528096" event={"ID":"f88e16ca-b62a-40a3-92b0-6102b5eeb91f","Type":"ContainerDied","Data":"a25afd14c697bdad643f7d9b68e9a92cb3d0b36560061004eec0ee73cf5e8588"}
Sep 06 19:47:37.332684 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.332652    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469528096"
Sep 06 19:47:37.349577 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.349519    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7636-140191469528096_f88e16ca-b62a-40a3-92b0-6102b5eeb91f/test-curl-7636-140191469528096/0.log"
Sep 06 19:47:37.370040 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.370007    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7636-140191469528096"]
Sep 06 19:47:37.371800 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.371779    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7636-140191469528096"]
Sep 06 19:47:37.460734 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.460634    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hchj7\" (UniqueName: \"kubernetes.io/projected/f88e16ca-b62a-40a3-92b0-6102b5eeb91f-kube-api-access-hchj7\") pod \"f88e16ca-b62a-40a3-92b0-6102b5eeb91f\" (UID: \"f88e16ca-b62a-40a3-92b0-6102b5eeb91f\") "
Sep 06 19:47:37.462762 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.462728    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88e16ca-b62a-40a3-92b0-6102b5eeb91f-kube-api-access-hchj7" (OuterVolumeSpecName: "kube-api-access-hchj7") pod "f88e16ca-b62a-40a3-92b0-6102b5eeb91f" (UID: "f88e16ca-b62a-40a3-92b0-6102b5eeb91f"). InnerVolumeSpecName "kube-api-access-hchj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:47:37.486272 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.486240    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7636-140191469528096"]
Sep 06 19:47:37.486649 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.486635    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f88e16ca-b62a-40a3-92b0-6102b5eeb91f" containerName="test-curl-7636-140191469528096"
Sep 06 19:47:37.486687 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.486651    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f88e16ca-b62a-40a3-92b0-6102b5eeb91f" containerName="test-curl-7636-140191469528096"
Sep 06 19:47:37.486741 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.486722    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f88e16ca-b62a-40a3-92b0-6102b5eeb91f" containerName="test-curl-7636-140191469528096"
Sep 06 19:47:37.489745 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.489728    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469528096"
Sep 06 19:47:37.493251 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.493223    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-140191469528096"]
Sep 06 19:47:37.561921 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.561888    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm2v9\" (UniqueName: \"kubernetes.io/projected/24591f00-2f09-45de-8521-3875fa0326a0-kube-api-access-rm2v9\") pod \"test-curl-7636-140191469528096\" (UID: \"24591f00-2f09-45de-8521-3875fa0326a0\") " pod="opendatahub/test-curl-7636-140191469528096"
Sep 06 19:47:37.562161 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.562141    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hchj7\" (UniqueName: \"kubernetes.io/projected/f88e16ca-b62a-40a3-92b0-6102b5eeb91f-kube-api-access-hchj7\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:47:37.663328 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.663283    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm2v9\" (UniqueName: \"kubernetes.io/projected/24591f00-2f09-45de-8521-3875fa0326a0-kube-api-access-rm2v9\") pod \"test-curl-7636-140191469528096\" (UID: \"24591f00-2f09-45de-8521-3875fa0326a0\") " pod="opendatahub/test-curl-7636-140191469528096"
Sep 06 19:47:37.670265 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.670232    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm2v9\" (UniqueName: \"kubernetes.io/projected/24591f00-2f09-45de-8521-3875fa0326a0-kube-api-access-rm2v9\") pod \"test-curl-7636-140191469528096\" (UID: \"24591f00-2f09-45de-8521-3875fa0326a0\") " pod="opendatahub/test-curl-7636-140191469528096"
Sep 06 19:47:37.800098 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:37.799988    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469528096"
Sep 06 19:47:38.068690 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:38.068662    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7636-140191469528096"]
Sep 06 19:47:38.070931 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:47:38.070908    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24591f00_2f09_45de_8521_3875fa0326a0.slice/crio-cc1e45edf23d03f5e6b4e676b19498bfd03a727ee7084898ad4e240ccdc862ea WatchSource:0}: Error finding container cc1e45edf23d03f5e6b4e676b19498bfd03a727ee7084898ad4e240ccdc862ea: Status 404 returned error can't find the container with id cc1e45edf23d03f5e6b4e676b19498bfd03a727ee7084898ad4e240ccdc862ea
Sep 06 19:47:38.194870 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:38.194829    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-140191469528096" event={"ID":"24591f00-2f09-45de-8521-3875fa0326a0","Type":"ContainerStarted","Data":"cc1e45edf23d03f5e6b4e676b19498bfd03a727ee7084898ad4e240ccdc862ea"}
Sep 06 19:47:38.196232 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:38.196211    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="032530fceeba653336b9a8a66cf336c2d4deb95fb74dea66833d1e5960336e2c"
Sep 06 19:47:38.196370 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:38.196235    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469528096"
Sep 06 19:47:38.207329 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:38.207302    2571 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7636-140191469528096" oldPodUID="f88e16ca-b62a-40a3-92b0-6102b5eeb91f" podUID="24591f00-2f09-45de-8521-3875fa0326a0"
Sep 06 19:47:38.918079 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:38.918004    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f88e16ca-b62a-40a3-92b0-6102b5eeb91f" path="/var/lib/kubelet/pods/f88e16ca-b62a-40a3-92b0-6102b5eeb91f/volumes"
Sep 06 19:47:39.204016 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:39.203889    2571 generic.go:356] "Generic (PLEG): container finished" podID="24591f00-2f09-45de-8521-3875fa0326a0" containerID="5a7cbb5779b8b2c6e619b1b2ea780eae424b0e86c06c18e4918cb64bf8df57a2" exitCode=0
Sep 06 19:47:39.204016 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:39.203999    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7636-140191469528096" event={"ID":"24591f00-2f09-45de-8521-3875fa0326a0","Type":"ContainerDied","Data":"5a7cbb5779b8b2c6e619b1b2ea780eae424b0e86c06c18e4918cb64bf8df57a2"}
Sep 06 19:47:40.344338 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:40.344309    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469528096"
Sep 06 19:47:40.362510 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:40.362484    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7636-140191469528096_24591f00-2f09-45de-8521-3875fa0326a0/test-curl-7636-140191469528096/0.log"
Sep 06 19:47:40.383463 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:40.383432    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7636-140191469528096"]
Sep 06 19:47:40.386978 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:40.386936    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7636-140191469528096"]
Sep 06 19:47:40.490128 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:40.489920    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm2v9\" (UniqueName: \"kubernetes.io/projected/24591f00-2f09-45de-8521-3875fa0326a0-kube-api-access-rm2v9\") pod \"24591f00-2f09-45de-8521-3875fa0326a0\" (UID: \"24591f00-2f09-45de-8521-3875fa0326a0\") "
Sep 06 19:47:40.492141 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:40.492110    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24591f00-2f09-45de-8521-3875fa0326a0-kube-api-access-rm2v9" (OuterVolumeSpecName: "kube-api-access-rm2v9") pod "24591f00-2f09-45de-8521-3875fa0326a0" (UID: "24591f00-2f09-45de-8521-3875fa0326a0"). InnerVolumeSpecName "kube-api-access-rm2v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:47:40.591734 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:40.591690    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm2v9\" (UniqueName: \"kubernetes.io/projected/24591f00-2f09-45de-8521-3875fa0326a0-kube-api-access-rm2v9\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:47:40.918231 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:40.918196    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24591f00-2f09-45de-8521-3875fa0326a0" path="/var/lib/kubelet/pods/24591f00-2f09-45de-8521-3875fa0326a0/volumes"
Sep 06 19:47:41.215726 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.215641    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7636-140191469528096"
Sep 06 19:47:41.215726 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.215685    2571 scope.go:117] "RemoveContainer" containerID="5a7cbb5779b8b2c6e619b1b2ea780eae424b0e86c06c18e4918cb64bf8df57a2"
Sep 06 19:47:41.244421 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.244384    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"]
Sep 06 19:47:41.244773 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.244759    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24591f00-2f09-45de-8521-3875fa0326a0" containerName="test-curl-7636-140191469528096"
Sep 06 19:47:41.244830 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.244774    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="24591f00-2f09-45de-8521-3875fa0326a0" containerName="test-curl-7636-140191469528096"
Sep 06 19:47:41.244898 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.244849    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="24591f00-2f09-45de-8521-3875fa0326a0" containerName="test-curl-7636-140191469528096"
Sep 06 19:47:41.249377 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.249356    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.252274 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.252233    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0936e642-openshift-default-dockercfg-8nrjk\""
Sep 06 19:47:41.255714 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.255691    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:41.255947 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.255883    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:41.259870 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.259832    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"]
Sep 06 19:47:41.270488 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.270454    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:41.400019 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.399977    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-envoy\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.400439 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.400079    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8afd16b5-d1e3-473a-b290-30f1cefd444d-istiod-ca-cert\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.400439 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.400156    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-workload-certs\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.400439 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.400198    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-data\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.400439 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.400230    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-workload-socket\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.400439 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.400368    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-credential-socket\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.400675 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.400439    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-podinfo\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.400675 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.400482    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbslw\" (UniqueName: \"kubernetes.io/projected/8afd16b5-d1e3-473a-b290-30f1cefd444d-kube-api-access-jbslw\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.407259 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.401040    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-token\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.501852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.501745    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-token\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.501852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.501799    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-envoy\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.502115 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.501854    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8afd16b5-d1e3-473a-b290-30f1cefd444d-istiod-ca-cert\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.502115 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.501905    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-workload-certs\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.502115 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.501944    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-data\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.502115 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.501991    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-workload-socket\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.502115 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.502021    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-credential-socket\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.502115 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.502063    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-podinfo\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.502115 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.502100    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbslw\" (UniqueName: \"kubernetes.io/projected/8afd16b5-d1e3-473a-b290-30f1cefd444d-kube-api-access-jbslw\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.502569 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.502406    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-workload-certs\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.502638 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.502597    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-credential-socket\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.502638 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.502613    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-data\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.502712 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.502635    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-workload-socket\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.502766 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.502736    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8afd16b5-d1e3-473a-b290-30f1cefd444d-istiod-ca-cert\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.504458 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.504439    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-envoy\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.504631 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.504610    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-podinfo\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.509296 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.509267    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-token\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.509399 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.509314    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbslw\" (UniqueName: \"kubernetes.io/projected/8afd16b5-d1e3-473a-b290-30f1cefd444d-kube-api-access-jbslw\") pod \"e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") " pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.563096 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.563052    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:41.842030 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.841915    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"]
Sep 06 19:47:41.844888 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:47:41.844857    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afd16b5_d1e3_473a_b290_30f1cefd444d.slice/crio-193db677bfbe438db7edddda6db18ee8ea898e31b61d25628b3e5bd4a6ddb41e WatchSource:0}: Error finding container 193db677bfbe438db7edddda6db18ee8ea898e31b61d25628b3e5bd4a6ddb41e: Status 404 returned error can't find the container with id 193db677bfbe438db7edddda6db18ee8ea898e31b61d25628b3e5bd4a6ddb41e
Sep 06 19:47:41.847349 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.847312    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:47:41.847431 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.847384    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:47:41.847431 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:41.847413    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:47:42.225375 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:42.225271    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25" event={"ID":"8afd16b5-d1e3-473a-b290-30f1cefd444d","Type":"ContainerStarted","Data":"21e4482c2abd1192d36fb434896c8a73c37fc11b34d3142883986f4b00a61add"}
Sep 06 19:47:42.225375 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:42.225335    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25" event={"ID":"8afd16b5-d1e3-473a-b290-30f1cefd444d","Type":"ContainerStarted","Data":"193db677bfbe438db7edddda6db18ee8ea898e31b61d25628b3e5bd4a6ddb41e"}
Sep 06 19:47:42.238079 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:42.238049    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:47:42.244285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:42.244234    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25" podStartSLOduration=1.244212133 podStartE2EDuration="1.244212133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:47:42.242158719 +0000 UTC m=+1225.812457070" watchObservedRunningTime="2026-09-06 19:47:42.244212133 +0000 UTC m=+1225.814510480"
Sep 06 19:47:42.564049 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:42.564011    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:42.569119 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:42.569094    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:43.232269 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:43.232235    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:43.233551 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:43.233532    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:47:45.243057 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.243026    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"]
Sep 06 19:47:45.247372 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.247348    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:45.250054 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.250025    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0936e642-serving-cert\""
Sep 06 19:47:45.260121 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.260091    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"]
Sep 06 19:47:45.340289 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.340247    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4222df8-0a5b-43d5-adbd-c2b9448e8698-metrics-certs\") pod \"maas-api-e2e-mt-0936e642-6cb77b499-v8q9g\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:45.340512 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.340301    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwdpl\" (UniqueName: \"kubernetes.io/projected/a4222df8-0a5b-43d5-adbd-c2b9448e8698-kube-api-access-pwdpl\") pod \"maas-api-e2e-mt-0936e642-6cb77b499-v8q9g\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:45.340512 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.340386    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4222df8-0a5b-43d5-adbd-c2b9448e8698-maas-api-tls\") pod \"maas-api-e2e-mt-0936e642-6cb77b499-v8q9g\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:45.340512 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.340463    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4222df8-0a5b-43d5-adbd-c2b9448e8698-openshift-service-ca\") pod \"maas-api-e2e-mt-0936e642-6cb77b499-v8q9g\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:45.441351 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.441315    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4222df8-0a5b-43d5-adbd-c2b9448e8698-openshift-service-ca\") pod \"maas-api-e2e-mt-0936e642-6cb77b499-v8q9g\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:45.441526 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.441457    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4222df8-0a5b-43d5-adbd-c2b9448e8698-metrics-certs\") pod \"maas-api-e2e-mt-0936e642-6cb77b499-v8q9g\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:45.441526 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.441489    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwdpl\" (UniqueName: \"kubernetes.io/projected/a4222df8-0a5b-43d5-adbd-c2b9448e8698-kube-api-access-pwdpl\") pod \"maas-api-e2e-mt-0936e642-6cb77b499-v8q9g\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:45.441526 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.441518    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4222df8-0a5b-43d5-adbd-c2b9448e8698-maas-api-tls\") pod \"maas-api-e2e-mt-0936e642-6cb77b499-v8q9g\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:45.441691 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:47:45.441648    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-serving-cert: secret "maas-api-e2e-mt-0936e642-serving-cert" not found
Sep 06 19:47:45.441740 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:47:45.441715    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4222df8-0a5b-43d5-adbd-c2b9448e8698-maas-api-tls podName:a4222df8-0a5b-43d5-adbd-c2b9448e8698 nodeName:}" failed. No retries permitted until 2026-09-06 19:47:45.941694535 +0000 UTC m=+1229.511992862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4222df8-0a5b-43d5-adbd-c2b9448e8698-maas-api-tls") pod "maas-api-e2e-mt-0936e642-6cb77b499-v8q9g" (UID: "a4222df8-0a5b-43d5-adbd-c2b9448e8698") : secret "maas-api-e2e-mt-0936e642-serving-cert" not found
Sep 06 19:47:45.442116 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.442097    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4222df8-0a5b-43d5-adbd-c2b9448e8698-openshift-service-ca\") pod \"maas-api-e2e-mt-0936e642-6cb77b499-v8q9g\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:45.443863 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.443844    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4222df8-0a5b-43d5-adbd-c2b9448e8698-metrics-certs\") pod \"maas-api-e2e-mt-0936e642-6cb77b499-v8q9g\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:45.449639 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.449613    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwdpl\" (UniqueName: \"kubernetes.io/projected/a4222df8-0a5b-43d5-adbd-c2b9448e8698-kube-api-access-pwdpl\") pod \"maas-api-e2e-mt-0936e642-6cb77b499-v8q9g\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:45.945478 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.945443    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4222df8-0a5b-43d5-adbd-c2b9448e8698-maas-api-tls\") pod \"maas-api-e2e-mt-0936e642-6cb77b499-v8q9g\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:45.947855 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:45.947834    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4222df8-0a5b-43d5-adbd-c2b9448e8698-maas-api-tls\") pod \"maas-api-e2e-mt-0936e642-6cb77b499-v8q9g\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:46.159779 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:46.159723    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:46.430855 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:46.430824    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"]
Sep 06 19:47:46.432574 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:47:46.432544    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4222df8_0a5b_43d5_adbd_c2b9448e8698.slice/crio-9bd7dbe08bf2a4c43a37b3c9e522cbee76318ba0030b8e54ece5f47511e8cf7a WatchSource:0}: Error finding container 9bd7dbe08bf2a4c43a37b3c9e522cbee76318ba0030b8e54ece5f47511e8cf7a: Status 404 returned error can't find the container with id 9bd7dbe08bf2a4c43a37b3c9e522cbee76318ba0030b8e54ece5f47511e8cf7a
Sep 06 19:47:47.260270 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:47.260188    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g" event={"ID":"a4222df8-0a5b-43d5-adbd-c2b9448e8698","Type":"ContainerStarted","Data":"d58b17314d60a5570691f7596b7acd5724b5767b5369b055c1ae471d88259433"}
Sep 06 19:47:47.260270 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:47.260239    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g" event={"ID":"a4222df8-0a5b-43d5-adbd-c2b9448e8698","Type":"ContainerStarted","Data":"9bd7dbe08bf2a4c43a37b3c9e522cbee76318ba0030b8e54ece5f47511e8cf7a"}
Sep 06 19:47:47.260469 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:47.260313    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:47.277424 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:47.277374    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g" podStartSLOduration=1.86092506 podStartE2EDuration="2.277359104s" totalImagesPullingTime="416.434044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:47:47.274631827 +0000 UTC m=+1230.844930186" watchObservedRunningTime="2026-09-06 19:47:47.277359104 +0000 UTC m=+1230.847657468"
Sep 06 19:47:50.441934 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:50.441893    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:47:53.271505 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:53.271474    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:47:59.274218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:59.274173    2571 patch_prober.go:28] interesting pod/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 06 19:47:59.274620 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:47:59.274240    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25" podUID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 06 19:48:00.336178 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:00.336145    2571 generic.go:356] "Generic (PLEG): container finished" podID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerID="21e4482c2abd1192d36fb434896c8a73c37fc11b34d3142883986f4b00a61add" exitCode=0
Sep 06 19:48:00.336647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:00.336225    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25" event={"ID":"8afd16b5-d1e3-473a-b290-30f1cefd444d","Type":"ContainerDied","Data":"21e4482c2abd1192d36fb434896c8a73c37fc11b34d3142883986f4b00a61add"}
Sep 06 19:48:00.336744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:00.336728    2571 scope.go:117] "RemoveContainer" containerID="21e4482c2abd1192d36fb434896c8a73c37fc11b34d3142883986f4b00a61add"
Sep 06 19:48:00.337832 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:00.337805    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:48:00.337887 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:00.337877    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:48:00.337922 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:00.337907    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:48:01.348422 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:01.348375    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25" event={"ID":"8afd16b5-d1e3-473a-b290-30f1cefd444d","Type":"ContainerStarted","Data":"32e86f8ee77fc32d5ac59e8ff8c419871f930d2f18782eb0bc5a465297ba60ef"}
Sep 06 19:48:01.563250 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:01.563212    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:48:01.563439 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:01.563416    2571 patch_prober.go:28] interesting pod/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 06 19:48:01.563490 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:01.563466    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25" podUID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 06 19:48:02.564033 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:02.563997    2571 patch_prober.go:28] interesting pod/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 06 19:48:02.564515 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:02.564064    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25" podUID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 06 19:48:03.563938 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:03.563901    2571 patch_prober.go:28] interesting pod/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 06 19:48:03.564156 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:03.564011    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25" podUID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 06 19:48:04.564235 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:04.564200    2571 patch_prober.go:28] interesting pod/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 06 19:48:04.564625 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:04.564259    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25" podUID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 06 19:48:05.563424 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:05.563380    2571 patch_prober.go:28] interesting pod/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 06 19:48:05.563603 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:05.563449    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25" podUID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 06 19:48:06.591429 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:06.591383    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:48:06.591903 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:06.591779    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:48:06.592725 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:06.592708    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:48:14.944864 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:14.944782    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"]
Sep 06 19:48:14.949622 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:14.949598    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:14.951943 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:14.951922    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-712616\"/\"default-dockercfg-mjc8d\""
Sep 06 19:48:14.952087 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:14.951928    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-712616\"/\"openshift-service-ca.crt\""
Sep 06 19:48:14.953108 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:14.953085    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-712616\"/\"kube-root-ca.crt\""
Sep 06 19:48:14.953308 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:14.953116    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-712616\"/\"auth-test-model-712616-kserve-self-signed-certs\""
Sep 06 19:48:14.956414 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:14.956390    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"]
Sep 06 19:48:15.041014 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.040943    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-model-cache\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.041194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.041086    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-tls-certs\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.041194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.041151    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-tmp-dir\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.041194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.041176    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b52j\" (UniqueName: \"kubernetes.io/projected/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-kube-api-access-7b52j\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.041337 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.041199    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-home\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.041337 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.041217    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-dshm\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.142435 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.142397    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-home\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.142435 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.142435    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-dshm\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.142660 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.142554    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-model-cache\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.142708 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.142678    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-tls-certs\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.142798 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.142784    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-tmp-dir\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.142858 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.142816    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b52j\" (UniqueName: \"kubernetes.io/projected/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-kube-api-access-7b52j\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.142916 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.142887    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-home\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.142986 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.142915    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-model-cache\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.143208 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.143187    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-tmp-dir\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.144686 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.144667    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-dshm\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.145055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.145038    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-tls-certs\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.149908 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.149878    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b52j\" (UniqueName: \"kubernetes.io/projected/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-kube-api-access-7b52j\") pod \"auth-test-model-712616-kserve-894fd44dc-l6lmj\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") " pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.260592 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.260499    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:15.537397 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:15.537367    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"]
Sep 06 19:48:15.539280 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:48:15.539251    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcc7a049_eb1e_4cc9_a39f_ca0e47affbe8.slice/crio-67a917735784ba21e9e8465a008a86d967f924a8ebd41fed4cfafa61b9ae13e4 WatchSource:0}: Error finding container 67a917735784ba21e9e8465a008a86d967f924a8ebd41fed4cfafa61b9ae13e4: Status 404 returned error can't find the container with id 67a917735784ba21e9e8465a008a86d967f924a8ebd41fed4cfafa61b9ae13e4
Sep 06 19:48:16.436052 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:16.436015    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj" event={"ID":"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8","Type":"ContainerStarted","Data":"fee18ee2a8f25920fe2db31c6a40b40caf93d198e9f9a1489e0438449121f9dc"}
Sep 06 19:48:16.436451 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:16.436060    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj" event={"ID":"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8","Type":"ContainerStarted","Data":"67a917735784ba21e9e8465a008a86d967f924a8ebd41fed4cfafa61b9ae13e4"}
Sep 06 19:48:16.451531 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:16.451480    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj" podStartSLOduration=2.451465065 podStartE2EDuration="2.451465065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:48:16.450447595 +0000 UTC m=+1260.020745941" watchObservedRunningTime="2026-09-06 19:48:16.451465065 +0000 UTC m=+1260.021763411"
Sep 06 19:48:17.059165 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:17.059131    2571 scope.go:117] "RemoveContainer" containerID="61d70076167515a14e09b2cabedd7c4a42dff163b28173ebb905cef2c0455049"
Sep 06 19:48:22.052041 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:22.052006    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:48:25.261417 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:25.261376    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:25.261417 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:25.261425    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:25.274388 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:25.274354    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:25.498732 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:25.498701    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:48:42.443726 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:48:42.443682    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:49:17.093068 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:49:17.093037    2571 scope.go:117] "RemoveContainer" containerID="430492765ce139275f138a43c6ad8feff017ab90ae3becdfbb65ae9bf4aeb917"
Sep 06 19:49:17.102083 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:49:17.102058    2571 scope.go:117] "RemoveContainer" containerID="c1aef4a9a532b51c891abcd5ba68cdd78f400728c5084e1f316d7fefa4bca0c2"
Sep 06 19:49:21.844223 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:49:21.844185    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:49:40.141768 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:49:40.141683    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:49:55.455215 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:49:55.455167    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:50:10.439432 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:10.439396    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"]
Sep 06 19:50:10.439860 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:10.439648    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g" podUID="a4222df8-0a5b-43d5-adbd-c2b9448e8698" containerName="maas-api" containerID="cri-o://d58b17314d60a5570691f7596b7acd5724b5767b5369b055c1ae471d88259433" gracePeriod=30
Sep 06 19:50:10.699881 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:10.699819    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:50:10.826917 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:10.826879    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwdpl\" (UniqueName: \"kubernetes.io/projected/a4222df8-0a5b-43d5-adbd-c2b9448e8698-kube-api-access-pwdpl\") pod \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") "
Sep 06 19:50:10.827455 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:10.827412    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4222df8-0a5b-43d5-adbd-c2b9448e8698-openshift-service-ca\") pod \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") "
Sep 06 19:50:10.827783 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:10.827739    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4222df8-0a5b-43d5-adbd-c2b9448e8698-metrics-certs\") pod \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") "
Sep 06 19:50:10.827878 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:10.827797    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4222df8-0a5b-43d5-adbd-c2b9448e8698-maas-api-tls\") pod \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\" (UID: \"a4222df8-0a5b-43d5-adbd-c2b9448e8698\") "
Sep 06 19:50:10.827943 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:10.827884    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4222df8-0a5b-43d5-adbd-c2b9448e8698-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4222df8-0a5b-43d5-adbd-c2b9448e8698" (UID: "a4222df8-0a5b-43d5-adbd-c2b9448e8698"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:50:10.830598 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:10.829621    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4222df8-0a5b-43d5-adbd-c2b9448e8698-openshift-service-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:50:10.834376 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:10.831018    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4222df8-0a5b-43d5-adbd-c2b9448e8698-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4222df8-0a5b-43d5-adbd-c2b9448e8698" (UID: "a4222df8-0a5b-43d5-adbd-c2b9448e8698"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:50:10.836194 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:10.836168    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4222df8-0a5b-43d5-adbd-c2b9448e8698-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a4222df8-0a5b-43d5-adbd-c2b9448e8698" (UID: "a4222df8-0a5b-43d5-adbd-c2b9448e8698"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:50:10.837306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:10.837281    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4222df8-0a5b-43d5-adbd-c2b9448e8698-kube-api-access-pwdpl" (OuterVolumeSpecName: "kube-api-access-pwdpl") pod "a4222df8-0a5b-43d5-adbd-c2b9448e8698" (UID: "a4222df8-0a5b-43d5-adbd-c2b9448e8698"). InnerVolumeSpecName "kube-api-access-pwdpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:50:10.931116 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:10.931085    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwdpl\" (UniqueName: \"kubernetes.io/projected/a4222df8-0a5b-43d5-adbd-c2b9448e8698-kube-api-access-pwdpl\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:50:10.931116 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:10.931117    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4222df8-0a5b-43d5-adbd-c2b9448e8698-metrics-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:50:10.931301 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:10.931130    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4222df8-0a5b-43d5-adbd-c2b9448e8698-maas-api-tls\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:50:11.053434 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:11.053342    2571 generic.go:356] "Generic (PLEG): container finished" podID="a4222df8-0a5b-43d5-adbd-c2b9448e8698" containerID="d58b17314d60a5570691f7596b7acd5724b5767b5369b055c1ae471d88259433" exitCode=0
Sep 06 19:50:11.053434 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:11.053390    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g" event={"ID":"a4222df8-0a5b-43d5-adbd-c2b9448e8698","Type":"ContainerDied","Data":"d58b17314d60a5570691f7596b7acd5724b5767b5369b055c1ae471d88259433"}
Sep 06 19:50:11.053434 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:11.053412    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"
Sep 06 19:50:11.053777 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:11.053444    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g" event={"ID":"a4222df8-0a5b-43d5-adbd-c2b9448e8698","Type":"ContainerDied","Data":"9bd7dbe08bf2a4c43a37b3c9e522cbee76318ba0030b8e54ece5f47511e8cf7a"}
Sep 06 19:50:11.053777 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:11.053474    2571 scope.go:117] "RemoveContainer" containerID="d58b17314d60a5570691f7596b7acd5724b5767b5369b055c1ae471d88259433"
Sep 06 19:50:11.062935 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:11.062913    2571 scope.go:117] "RemoveContainer" containerID="d58b17314d60a5570691f7596b7acd5724b5767b5369b055c1ae471d88259433"
Sep 06 19:50:11.063269 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:50:11.063241    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58b17314d60a5570691f7596b7acd5724b5767b5369b055c1ae471d88259433\": container with ID starting with d58b17314d60a5570691f7596b7acd5724b5767b5369b055c1ae471d88259433 not found: ID does not exist" containerID="d58b17314d60a5570691f7596b7acd5724b5767b5369b055c1ae471d88259433"
Sep 06 19:50:11.063323 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:11.063279    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58b17314d60a5570691f7596b7acd5724b5767b5369b055c1ae471d88259433"} err="failed to get container status \"d58b17314d60a5570691f7596b7acd5724b5767b5369b055c1ae471d88259433\": rpc error: code = NotFound desc = could not find container \"d58b17314d60a5570691f7596b7acd5724b5767b5369b055c1ae471d88259433\": container with ID starting with d58b17314d60a5570691f7596b7acd5724b5767b5369b055c1ae471d88259433 not found: ID does not exist"
Sep 06 19:50:11.068863 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:11.068833    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"]
Sep 06 19:50:11.072269 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:11.072245    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0936e642-6cb77b499-v8q9g"]
Sep 06 19:50:12.635525 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:12.635486    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:50:12.918779 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:12.918697    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4222df8-0a5b-43d5-adbd-c2b9448e8698" path="/var/lib/kubelet/pods/a4222df8-0a5b-43d5-adbd-c2b9448e8698/volumes"
Sep 06 19:50:17.128043 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:17.128007    2571 scope.go:117] "RemoveContainer" containerID="a57c065a26948d3e47454186ece45cb98b33104c609595675785a6348faa4517"
Sep 06 19:50:17.136678 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:17.136655    2571 scope.go:117] "RemoveContainer" containerID="0cafe4b673068ea14a922ff399f35ab41b36edcef3cfe506128d2340fcec3384"
Sep 06 19:50:27.040621 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:27.040575    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:50:43.334919 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:50:43.334879    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:51:00.034835 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:00.034800    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:51:08.242004 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:08.241952    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"]
Sep 06 19:51:09.459747 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:09.459663    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"]
Sep 06 19:51:16.445266 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:16.445225    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:51:17.163882 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:17.163848    2571 scope.go:117] "RemoveContainer" containerID="8223c44907d6fd8e905943bf8dc44a602085195613b04ed23d4ef5de2540bad8"
Sep 06 19:51:18.337439 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:18.337399    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"]
Sep 06 19:51:18.337840 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:18.337701    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25" podUID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerName="istio-proxy" containerID="cri-o://32e86f8ee77fc32d5ac59e8ff8c419871f930d2f18782eb0bc5a465297ba60ef" gracePeriod=30
Sep 06 19:51:21.594576 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:21.594539    2571 patch_prober.go:28] interesting pod/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 06 19:51:21.594980 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:21.594593    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25" podUID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 19:51:23.262484 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.262420    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7" podUID="c6e8086c-3282-464e-bc93-937a3aa06ab0" containerName="main" containerID="cri-o://519c5549445c2a205c7e4650be8d9c840349188972da4bd4ac23171bc843d8cd" gracePeriod=45
Sep 06 19:51:23.460044 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.460007    2571 generic.go:356] "Generic (PLEG): container finished" podID="c6e8086c-3282-464e-bc93-937a3aa06ab0" containerID="519c5549445c2a205c7e4650be8d9c840349188972da4bd4ac23171bc843d8cd" exitCode=0
Sep 06 19:51:23.460208 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.460078    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7" event={"ID":"c6e8086c-3282-464e-bc93-937a3aa06ab0","Type":"ContainerDied","Data":"519c5549445c2a205c7e4650be8d9c840349188972da4bd4ac23171bc843d8cd"}
Sep 06 19:51:23.462540 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.462510    2571 generic.go:356] "Generic (PLEG): container finished" podID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerID="32e86f8ee77fc32d5ac59e8ff8c419871f930d2f18782eb0bc5a465297ba60ef" exitCode=0
Sep 06 19:51:23.462690 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.462586    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25" event={"ID":"8afd16b5-d1e3-473a-b290-30f1cefd444d","Type":"ContainerDied","Data":"32e86f8ee77fc32d5ac59e8ff8c419871f930d2f18782eb0bc5a465297ba60ef"}
Sep 06 19:51:23.462690 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.462670    2571 scope.go:117] "RemoveContainer" containerID="21e4482c2abd1192d36fb434896c8a73c37fc11b34d3142883986f4b00a61add"
Sep 06 19:51:23.618335 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.618309    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:51:23.620334 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.620313    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:51:23.772354 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.772318    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8bv5\" (UniqueName: \"kubernetes.io/projected/c6e8086c-3282-464e-bc93-937a3aa06ab0-kube-api-access-t8bv5\") pod \"c6e8086c-3282-464e-bc93-937a3aa06ab0\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") "
Sep 06 19:51:23.772354 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.772361    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e8086c-3282-464e-bc93-937a3aa06ab0-tls-certs\") pod \"c6e8086c-3282-464e-bc93-937a3aa06ab0\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") "
Sep 06 19:51:23.772563 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.772385    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-dshm\") pod \"c6e8086c-3282-464e-bc93-937a3aa06ab0\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") "
Sep 06 19:51:23.772563 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.772409    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8afd16b5-d1e3-473a-b290-30f1cefd444d-istiod-ca-cert\") pod \"8afd16b5-d1e3-473a-b290-30f1cefd444d\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") "
Sep 06 19:51:23.772563 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.772444    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-data\") pod \"8afd16b5-d1e3-473a-b290-30f1cefd444d\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") "
Sep 06 19:51:23.772693 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.772603    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-credential-socket\") pod \"8afd16b5-d1e3-473a-b290-30f1cefd444d\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") "
Sep 06 19:51:23.772693 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.772644    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbslw\" (UniqueName: \"kubernetes.io/projected/8afd16b5-d1e3-473a-b290-30f1cefd444d-kube-api-access-jbslw\") pod \"8afd16b5-d1e3-473a-b290-30f1cefd444d\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") "
Sep 06 19:51:23.772693 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.772676    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-workload-socket\") pod \"8afd16b5-d1e3-473a-b290-30f1cefd444d\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") "
Sep 06 19:51:23.772857 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.772714    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-podinfo\") pod \"8afd16b5-d1e3-473a-b290-30f1cefd444d\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") "
Sep 06 19:51:23.772857 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.772772    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8afd16b5-d1e3-473a-b290-30f1cefd444d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8afd16b5-d1e3-473a-b290-30f1cefd444d" (UID: "8afd16b5-d1e3-473a-b290-30f1cefd444d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:51:23.772857 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.772786    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-tmp-dir\") pod \"c6e8086c-3282-464e-bc93-937a3aa06ab0\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") "
Sep 06 19:51:23.772857 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.772834    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-model-cache\") pod \"c6e8086c-3282-464e-bc93-937a3aa06ab0\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") "
Sep 06 19:51:23.773115 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.772888    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-token\") pod \"8afd16b5-d1e3-473a-b290-30f1cefd444d\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") "
Sep 06 19:51:23.773115 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.772891    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8afd16b5-d1e3-473a-b290-30f1cefd444d" (UID: "8afd16b5-d1e3-473a-b290-30f1cefd444d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:51:23.773115 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.772998    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6e8086c-3282-464e-bc93-937a3aa06ab0" (UID: "c6e8086c-3282-464e-bc93-937a3aa06ab0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:51:23.773115 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.773014    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-data" (OuterVolumeSpecName: "istio-data") pod "8afd16b5-d1e3-473a-b290-30f1cefd444d" (UID: "8afd16b5-d1e3-473a-b290-30f1cefd444d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:51:23.773115 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.773020    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-home\") pod \"c6e8086c-3282-464e-bc93-937a3aa06ab0\" (UID: \"c6e8086c-3282-464e-bc93-937a3aa06ab0\") "
Sep 06 19:51:23.773115 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.773072    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-envoy\") pod \"8afd16b5-d1e3-473a-b290-30f1cefd444d\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") "
Sep 06 19:51:23.773437 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.773163    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-workload-certs\") pod \"8afd16b5-d1e3-473a-b290-30f1cefd444d\" (UID: \"8afd16b5-d1e3-473a-b290-30f1cefd444d\") "
Sep 06 19:51:23.773437 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.773197    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8afd16b5-d1e3-473a-b290-30f1cefd444d" (UID: "8afd16b5-d1e3-473a-b290-30f1cefd444d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:51:23.773437 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.773218    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-home" (OuterVolumeSpecName: "home") pod "c6e8086c-3282-464e-bc93-937a3aa06ab0" (UID: "c6e8086c-3282-464e-bc93-937a3aa06ab0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:51:23.773640 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.773620    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-home\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:23.773705 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.773645    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8afd16b5-d1e3-473a-b290-30f1cefd444d-istiod-ca-cert\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:23.773705 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.773660    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-data\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:23.773705 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.773672    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-credential-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:23.773705 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.773683    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-workload-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:23.773705 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.773697    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-tmp-dir\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:23.773988 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.773938    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8afd16b5-d1e3-473a-b290-30f1cefd444d" (UID: "8afd16b5-d1e3-473a-b290-30f1cefd444d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:51:23.774154 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.774132    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6e8086c-3282-464e-bc93-937a3aa06ab0" (UID: "c6e8086c-3282-464e-bc93-937a3aa06ab0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:51:23.775067 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.775030    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-dshm" (OuterVolumeSpecName: "dshm") pod "c6e8086c-3282-464e-bc93-937a3aa06ab0" (UID: "c6e8086c-3282-464e-bc93-937a3aa06ab0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:51:23.775202 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.775174    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6e8086c-3282-464e-bc93-937a3aa06ab0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6e8086c-3282-464e-bc93-937a3aa06ab0" (UID: "c6e8086c-3282-464e-bc93-937a3aa06ab0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:51:23.775478 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.775455    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8afd16b5-d1e3-473a-b290-30f1cefd444d" (UID: "8afd16b5-d1e3-473a-b290-30f1cefd444d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:51:23.775568 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.775500    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8afd16b5-d1e3-473a-b290-30f1cefd444d" (UID: "8afd16b5-d1e3-473a-b290-30f1cefd444d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 19:51:23.775568 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.775508    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-token" (OuterVolumeSpecName: "istio-token") pod "8afd16b5-d1e3-473a-b290-30f1cefd444d" (UID: "8afd16b5-d1e3-473a-b290-30f1cefd444d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:51:23.775568 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.775522    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e8086c-3282-464e-bc93-937a3aa06ab0-kube-api-access-t8bv5" (OuterVolumeSpecName: "kube-api-access-t8bv5") pod "c6e8086c-3282-464e-bc93-937a3aa06ab0" (UID: "c6e8086c-3282-464e-bc93-937a3aa06ab0"). InnerVolumeSpecName "kube-api-access-t8bv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:51:23.776231 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.776212    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8afd16b5-d1e3-473a-b290-30f1cefd444d-kube-api-access-jbslw" (OuterVolumeSpecName: "kube-api-access-jbslw") pod "8afd16b5-d1e3-473a-b290-30f1cefd444d" (UID: "8afd16b5-d1e3-473a-b290-30f1cefd444d"). InnerVolumeSpecName "kube-api-access-jbslw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:51:23.875328 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.875277    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-envoy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:23.875328 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.875330    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8afd16b5-d1e3-473a-b290-30f1cefd444d-workload-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:23.875328 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.875341    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8bv5\" (UniqueName: \"kubernetes.io/projected/c6e8086c-3282-464e-bc93-937a3aa06ab0-kube-api-access-t8bv5\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:23.875588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.875353    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e8086c-3282-464e-bc93-937a3aa06ab0-tls-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:23.875588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.875362    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-dshm\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:23.875588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.875371    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbslw\" (UniqueName: \"kubernetes.io/projected/8afd16b5-d1e3-473a-b290-30f1cefd444d-kube-api-access-jbslw\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:23.875588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.875380    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-podinfo\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:23.875588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.875387    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e8086c-3282-464e-bc93-937a3aa06ab0-model-cache\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:23.875588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:23.875395    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8afd16b5-d1e3-473a-b290-30f1cefd444d-istio-token\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:24.470244 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.469974    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25" event={"ID":"8afd16b5-d1e3-473a-b290-30f1cefd444d","Type":"ContainerDied","Data":"193db677bfbe438db7edddda6db18ee8ea898e31b61d25628b3e5bd4a6ddb41e"}
Sep 06 19:51:24.470244 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.470046    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"
Sep 06 19:51:24.470244 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.470099    2571 scope.go:117] "RemoveContainer" containerID="32e86f8ee77fc32d5ac59e8ff8c419871f930d2f18782eb0bc5a465297ba60ef"
Sep 06 19:51:24.478826 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.478766    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7" event={"ID":"c6e8086c-3282-464e-bc93-937a3aa06ab0","Type":"ContainerDied","Data":"6663f2ac24bace297c75680a95bc1c710b757ce1db3876cdc8696adc6514d7f0"}
Sep 06 19:51:24.478826 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.478796    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"
Sep 06 19:51:24.479842 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.479811    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj" podUID="dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8" containerName="main" containerID="cri-o://fee18ee2a8f25920fe2db31c6a40b40caf93d198e9f9a1489e0438449121f9dc" gracePeriod=45
Sep 06 19:51:24.488370 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.488246    2571 scope.go:117] "RemoveContainer" containerID="519c5549445c2a205c7e4650be8d9c840349188972da4bd4ac23171bc843d8cd"
Sep 06 19:51:24.496734 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.496512    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"]
Sep 06 19:51:24.499062 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.499039    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0936e642-openshift-default-7b4cf5fdc6-s7l25"]
Sep 06 19:51:24.555429 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.555327    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"]
Sep 06 19:51:24.558858 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.558831    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b571c4/auth-test-model-b571c4-kserve-66b78f66c5-r6gr7"]
Sep 06 19:51:24.733729 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.733703    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:51:24.886529 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.886479    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-tmp-dir\") pod \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") "
Sep 06 19:51:24.886529 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.886532    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7b52j\" (UniqueName: \"kubernetes.io/projected/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-kube-api-access-7b52j\") pod \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") "
Sep 06 19:51:24.886790 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.886607    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-tls-certs\") pod \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") "
Sep 06 19:51:24.886790 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.886672    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-home\") pod \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") "
Sep 06 19:51:24.886790 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.886719    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-model-cache\") pod \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") "
Sep 06 19:51:24.886790 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.886749    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-dshm\") pod \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\" (UID: \"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8\") "
Sep 06 19:51:24.887036 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.886820    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8" (UID: "dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:51:24.887036 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.887011    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-home" (OuterVolumeSpecName: "home") pod "dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8" (UID: "dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:51:24.887144 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.887058    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-model-cache" (OuterVolumeSpecName: "model-cache") pod "dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8" (UID: "dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:51:24.887204 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.887171    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-tmp-dir\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:24.887204 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.887191    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-home\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:24.887278 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.887204    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-model-cache\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:24.889061 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.889035    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-dshm" (OuterVolumeSpecName: "dshm") pod "dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8" (UID: "dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:51:24.889191 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.889060    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8" (UID: "dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:51:24.889191 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.889074    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-kube-api-access-7b52j" (OuterVolumeSpecName: "kube-api-access-7b52j") pod "dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8" (UID: "dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8"). InnerVolumeSpecName "kube-api-access-7b52j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:51:24.919073 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.919032    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8afd16b5-d1e3-473a-b290-30f1cefd444d" path="/var/lib/kubelet/pods/8afd16b5-d1e3-473a-b290-30f1cefd444d/volumes"
Sep 06 19:51:24.919546 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.919525    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6e8086c-3282-464e-bc93-937a3aa06ab0" path="/var/lib/kubelet/pods/c6e8086c-3282-464e-bc93-937a3aa06ab0/volumes"
Sep 06 19:51:24.988375 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.988336    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-tls-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:24.988375 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.988363    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-dshm\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:24.988375 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:24.988373    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b52j\" (UniqueName: \"kubernetes.io/projected/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8-kube-api-access-7b52j\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:51:25.488360 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:25.488324    2571 generic.go:356] "Generic (PLEG): container finished" podID="dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8" containerID="fee18ee2a8f25920fe2db31c6a40b40caf93d198e9f9a1489e0438449121f9dc" exitCode=0
Sep 06 19:51:25.488360 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:25.488360    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"
Sep 06 19:51:25.488920 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:25.488407    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj" event={"ID":"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8","Type":"ContainerDied","Data":"fee18ee2a8f25920fe2db31c6a40b40caf93d198e9f9a1489e0438449121f9dc"}
Sep 06 19:51:25.488920 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:25.488458    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj" event={"ID":"dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8","Type":"ContainerDied","Data":"67a917735784ba21e9e8465a008a86d967f924a8ebd41fed4cfafa61b9ae13e4"}
Sep 06 19:51:25.488920 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:25.488483    2571 scope.go:117] "RemoveContainer" containerID="fee18ee2a8f25920fe2db31c6a40b40caf93d198e9f9a1489e0438449121f9dc"
Sep 06 19:51:25.497114 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:25.497093    2571 scope.go:117] "RemoveContainer" containerID="fee18ee2a8f25920fe2db31c6a40b40caf93d198e9f9a1489e0438449121f9dc"
Sep 06 19:51:25.497407 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:51:25.497387    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee18ee2a8f25920fe2db31c6a40b40caf93d198e9f9a1489e0438449121f9dc\": container with ID starting with fee18ee2a8f25920fe2db31c6a40b40caf93d198e9f9a1489e0438449121f9dc not found: ID does not exist" containerID="fee18ee2a8f25920fe2db31c6a40b40caf93d198e9f9a1489e0438449121f9dc"
Sep 06 19:51:25.497492 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:25.497416    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee18ee2a8f25920fe2db31c6a40b40caf93d198e9f9a1489e0438449121f9dc"} err="failed to get container status \"fee18ee2a8f25920fe2db31c6a40b40caf93d198e9f9a1489e0438449121f9dc\": rpc error: code = NotFound desc = could not find container \"fee18ee2a8f25920fe2db31c6a40b40caf93d198e9f9a1489e0438449121f9dc\": container with ID starting with fee18ee2a8f25920fe2db31c6a40b40caf93d198e9f9a1489e0438449121f9dc not found: ID does not exist"
Sep 06 19:51:25.504222 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:25.504198    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"]
Sep 06 19:51:25.508091 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:25.508069    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/auth-test-model-712616-kserve-894fd44dc-l6lmj"]
Sep 06 19:51:26.919081 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:26.919047    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8" path="/var/lib/kubelet/pods/dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8/volumes"
Sep 06 19:51:33.958952 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.958912    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"]
Sep 06 19:51:33.959480 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.959459    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8" containerName="main"
Sep 06 19:51:33.959534 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.959484    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8" containerName="main"
Sep 06 19:51:33.959534 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.959509    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerName="istio-proxy"
Sep 06 19:51:33.959534 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.959519    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerName="istio-proxy"
Sep 06 19:51:33.959636 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.959536    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e8086c-3282-464e-bc93-937a3aa06ab0" containerName="main"
Sep 06 19:51:33.959636 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.959545    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e8086c-3282-464e-bc93-937a3aa06ab0" containerName="main"
Sep 06 19:51:33.959636 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.959558    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerName="istio-proxy"
Sep 06 19:51:33.959636 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.959565    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerName="istio-proxy"
Sep 06 19:51:33.959636 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.959574    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4222df8-0a5b-43d5-adbd-c2b9448e8698" containerName="maas-api"
Sep 06 19:51:33.959636 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.959581    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4222df8-0a5b-43d5-adbd-c2b9448e8698" containerName="maas-api"
Sep 06 19:51:33.959834 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.959663    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerName="istio-proxy"
Sep 06 19:51:33.959834 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.959677    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8afd16b5-d1e3-473a-b290-30f1cefd444d" containerName="istio-proxy"
Sep 06 19:51:33.959834 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.959685    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4222df8-0a5b-43d5-adbd-c2b9448e8698" containerName="maas-api"
Sep 06 19:51:33.959834 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.959692    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcc7a049-eb1e-4cc9-a39f-ca0e47affbe8" containerName="main"
Sep 06 19:51:33.959834 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.959706    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6e8086c-3282-464e-bc93-937a3aa06ab0" containerName="main"
Sep 06 19:51:33.964688 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.964656    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:33.968088 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.968064    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-712616\"/\"sub-test-model-712616-kserve-self-signed-certs\""
Sep 06 19:51:33.969109 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.969092    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-712616\"/\"kube-root-ca.crt\""
Sep 06 19:51:33.969448 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.969429    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-712616\"/\"default-dockercfg-mjc8d\""
Sep 06 19:51:33.969541 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.969469    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-712616\"/\"openshift-service-ca.crt\""
Sep 06 19:51:33.973997 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:33.973973    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"]
Sep 06 19:51:34.079038 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.078997    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-home\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.079242 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.079053    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-model-cache\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.079242 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.079097    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-dshm\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.079242 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.079136    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-tmp-dir\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.079242 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.079198    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khq26\" (UniqueName: \"kubernetes.io/projected/8e86f22f-7bfd-42c9-a713-d09c80183cd1-kube-api-access-khq26\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.079242 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.079230    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e86f22f-7bfd-42c9-a713-d09c80183cd1-tls-certs\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.180445 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.180410    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-dshm\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.180638 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.180474    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-tmp-dir\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.180638 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.180516    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khq26\" (UniqueName: \"kubernetes.io/projected/8e86f22f-7bfd-42c9-a713-d09c80183cd1-kube-api-access-khq26\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.180638 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.180539    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e86f22f-7bfd-42c9-a713-d09c80183cd1-tls-certs\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.180638 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.180629    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-home\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.180874 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.180668    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-model-cache\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.182141 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.182110    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-model-cache\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.182141 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.182129    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-home\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.183213 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.182811    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-dshm\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.183213 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.183136    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-tmp-dir\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.185760 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.185727    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e86f22f-7bfd-42c9-a713-d09c80183cd1-tls-certs\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.194744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.194716    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khq26\" (UniqueName: \"kubernetes.io/projected/8e86f22f-7bfd-42c9-a713-d09c80183cd1-kube-api-access-khq26\") pod \"sub-test-model-712616-kserve-fb6d8cbb9-hknxn\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") " pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.274630 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.274542    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:34.546493 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:34.546467    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"]
Sep 06 19:51:34.548661 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:51:34.548573    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e86f22f_7bfd_42c9_a713_d09c80183cd1.slice/crio-f1c5060777a7e92e466bc73ea0bab2fbf0a30d4f3fb6fe336a2f6b45f7c198a8 WatchSource:0}: Error finding container f1c5060777a7e92e466bc73ea0bab2fbf0a30d4f3fb6fe336a2f6b45f7c198a8: Status 404 returned error can't find the container with id f1c5060777a7e92e466bc73ea0bab2fbf0a30d4f3fb6fe336a2f6b45f7c198a8
Sep 06 19:51:35.546085 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:35.546043    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn" event={"ID":"8e86f22f-7bfd-42c9-a713-d09c80183cd1","Type":"ContainerStarted","Data":"747f51563c62c6206ae9bba2bdfdb5d0f17eec3cabd865aa767da79bc9dac9af"}
Sep 06 19:51:35.546466 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:35.546093    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn" event={"ID":"8e86f22f-7bfd-42c9-a713-d09c80183cd1","Type":"ContainerStarted","Data":"f1c5060777a7e92e466bc73ea0bab2fbf0a30d4f3fb6fe336a2f6b45f7c198a8"}
Sep 06 19:51:35.576420 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:35.576361    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn" podStartSLOduration=2.57634526 podStartE2EDuration="2.57634526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:51:35.574206756 +0000 UTC m=+1459.144505103" watchObservedRunningTime="2026-09-06 19:51:35.57634526 +0000 UTC m=+1459.146643607"
Sep 06 19:51:37.737484 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:37.737445    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:51:44.274789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:44.274754    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:44.275292 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:44.274839    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:44.287395 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:44.287368    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:44.602926 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:44.602897    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:51:58.037934 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:51:58.037898    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:52:08.574368 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:08.574332    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7639-fd4a6a"]
Sep 06 19:52:08.578063 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:08.578043    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7639-fd4a6a"
Sep 06 19:52:08.580397 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:08.580371    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kn2d6\""
Sep 06 19:52:08.581026 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:08.581002    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7639-fd4a6a"]
Sep 06 19:52:08.712906 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:08.712872    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqwdr\" (UniqueName: \"kubernetes.io/projected/042955d3-46d1-4fd5-bfd0-cd5056a47fa2-kube-api-access-vqwdr\") pod \"mt-curl-7639-fd4a6a\" (UID: \"042955d3-46d1-4fd5-bfd0-cd5056a47fa2\") " pod="opendatahub/mt-curl-7639-fd4a6a"
Sep 06 19:52:08.814182 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:08.814150    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqwdr\" (UniqueName: \"kubernetes.io/projected/042955d3-46d1-4fd5-bfd0-cd5056a47fa2-kube-api-access-vqwdr\") pod \"mt-curl-7639-fd4a6a\" (UID: \"042955d3-46d1-4fd5-bfd0-cd5056a47fa2\") " pod="opendatahub/mt-curl-7639-fd4a6a"
Sep 06 19:52:08.821763 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:08.821736    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqwdr\" (UniqueName: \"kubernetes.io/projected/042955d3-46d1-4fd5-bfd0-cd5056a47fa2-kube-api-access-vqwdr\") pod \"mt-curl-7639-fd4a6a\" (UID: \"042955d3-46d1-4fd5-bfd0-cd5056a47fa2\") " pod="opendatahub/mt-curl-7639-fd4a6a"
Sep 06 19:52:08.888659 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:08.888556    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7639-fd4a6a"
Sep 06 19:52:09.156594 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:09.156518    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7639-fd4a6a"]
Sep 06 19:52:09.160126 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:52:09.160087    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod042955d3_46d1_4fd5_bfd0_cd5056a47fa2.slice/crio-3fe9089c658fb70d0831a7c9919d5e1ef074c88377294bcca194ae4614e4d641 WatchSource:0}: Error finding container 3fe9089c658fb70d0831a7c9919d5e1ef074c88377294bcca194ae4614e4d641: Status 404 returned error can't find the container with id 3fe9089c658fb70d0831a7c9919d5e1ef074c88377294bcca194ae4614e4d641
Sep 06 19:52:09.161600 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:09.161580    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 19:52:09.723687 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:09.723653    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7639-fd4a6a" event={"ID":"042955d3-46d1-4fd5-bfd0-cd5056a47fa2","Type":"ContainerStarted","Data":"3fe9089c658fb70d0831a7c9919d5e1ef074c88377294bcca194ae4614e4d641"}
Sep 06 19:52:10.737418 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:10.737379    2571 generic.go:356] "Generic (PLEG): container finished" podID="042955d3-46d1-4fd5-bfd0-cd5056a47fa2" containerID="26a79a4cecf9861e128364b9357e9f5f6446280fe34faf9ff100eff0e7b94127" exitCode=0
Sep 06 19:52:10.737868 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:10.737462    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7639-fd4a6a" event={"ID":"042955d3-46d1-4fd5-bfd0-cd5056a47fa2","Type":"ContainerDied","Data":"26a79a4cecf9861e128364b9357e9f5f6446280fe34faf9ff100eff0e7b94127"}
Sep 06 19:52:11.587376 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.587339    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"]
Sep 06 19:52:11.591337 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.591317    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.593891 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.593863    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cf16ea-openshift-default-dockercfg-s9bkx\""
Sep 06 19:52:11.601625 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.601597    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"]
Sep 06 19:52:11.746747 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.746715    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-istio-envoy\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.747099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.746760    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-credential-socket\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.747099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.746788    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7tr6\" (UniqueName: \"kubernetes.io/projected/ce637caa-427f-47a6-af77-6efff8fe023e-kube-api-access-f7tr6\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.747099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.746818    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-istio-data\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.747099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.746845    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-workload-certs\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.747099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.746923    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce637caa-427f-47a6-af77-6efff8fe023e-istiod-ca-cert\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.747099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.746986    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-workload-socket\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.747099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.747048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce637caa-427f-47a6-af77-6efff8fe023e-istio-token\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.747099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.747098    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce637caa-427f-47a6-af77-6efff8fe023e-istio-podinfo\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.848678 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.848597    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-istio-envoy\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.848678 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.848645    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-credential-socket\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.848912 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.848687    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7tr6\" (UniqueName: \"kubernetes.io/projected/ce637caa-427f-47a6-af77-6efff8fe023e-kube-api-access-f7tr6\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.848912 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.848719    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-istio-data\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.848912 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.848755    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-workload-certs\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.849138 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.848969    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce637caa-427f-47a6-af77-6efff8fe023e-istiod-ca-cert\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.849138 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.849026    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-workload-socket\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.849138 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.849067    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-credential-socket\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.849138 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.849078    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce637caa-427f-47a6-af77-6efff8fe023e-istio-token\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.849138 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.849126    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce637caa-427f-47a6-af77-6efff8fe023e-istio-podinfo\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.849350 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.849160    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-istio-data\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.849399 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.849339    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-workload-socket\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.849505 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.849488    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-workload-certs\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.849798 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.849769    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce637caa-427f-47a6-af77-6efff8fe023e-istiod-ca-cert\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.851363 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.851340    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-istio-envoy\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.851682 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.851662    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce637caa-427f-47a6-af77-6efff8fe023e-istio-podinfo\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.857634 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.857477    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce637caa-427f-47a6-af77-6efff8fe023e-istio-token\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.857796 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.857774    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7tr6\" (UniqueName: \"kubernetes.io/projected/ce637caa-427f-47a6-af77-6efff8fe023e-kube-api-access-f7tr6\") pod \"e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") " pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.880518 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.880494    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7639-fd4a6a"
Sep 06 19:52:11.903919 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.903888    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:11.908815 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.908790    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7639-fd4a6a_042955d3-46d1-4fd5-bfd0-cd5056a47fa2/mt-curl-7639-fd4a6a/0.log"
Sep 06 19:52:11.929210 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.929179    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7639-fd4a6a"]
Sep 06 19:52:11.932481 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:11.932450    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7639-fd4a6a"]
Sep 06 19:52:12.038748 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.038708    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7639-b308da"]
Sep 06 19:52:12.039147 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.039133    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="042955d3-46d1-4fd5-bfd0-cd5056a47fa2" containerName="mt-curl-7639-fd4a6a"
Sep 06 19:52:12.039210 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.039149    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="042955d3-46d1-4fd5-bfd0-cd5056a47fa2" containerName="mt-curl-7639-fd4a6a"
Sep 06 19:52:12.039255 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.039231    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="042955d3-46d1-4fd5-bfd0-cd5056a47fa2" containerName="mt-curl-7639-fd4a6a"
Sep 06 19:52:12.042635 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.042617    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7639-b308da"
Sep 06 19:52:12.048269 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.048245    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7639-b308da"]
Sep 06 19:52:12.050379 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.050347    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqwdr\" (UniqueName: \"kubernetes.io/projected/042955d3-46d1-4fd5-bfd0-cd5056a47fa2-kube-api-access-vqwdr\") pod \"042955d3-46d1-4fd5-bfd0-cd5056a47fa2\" (UID: \"042955d3-46d1-4fd5-bfd0-cd5056a47fa2\") "
Sep 06 19:52:12.053111 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.053086    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042955d3-46d1-4fd5-bfd0-cd5056a47fa2-kube-api-access-vqwdr" (OuterVolumeSpecName: "kube-api-access-vqwdr") pod "042955d3-46d1-4fd5-bfd0-cd5056a47fa2" (UID: "042955d3-46d1-4fd5-bfd0-cd5056a47fa2"). InnerVolumeSpecName "kube-api-access-vqwdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:52:12.152212 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.152118    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h8xk\" (UniqueName: \"kubernetes.io/projected/d71087d1-64c2-4d4e-80ea-33081476865a-kube-api-access-6h8xk\") pod \"mt-curl-7639-b308da\" (UID: \"d71087d1-64c2-4d4e-80ea-33081476865a\") " pod="opendatahub/mt-curl-7639-b308da"
Sep 06 19:52:12.152369 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.152219    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqwdr\" (UniqueName: \"kubernetes.io/projected/042955d3-46d1-4fd5-bfd0-cd5056a47fa2-kube-api-access-vqwdr\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:52:12.177620 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.177590    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"]
Sep 06 19:52:12.179032 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:52:12.179006    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce637caa_427f_47a6_af77_6efff8fe023e.slice/crio-26d08c8357b740756ccc6a98a8ada865802b68bd21fb705c143f8fb29ff9d1af WatchSource:0}: Error finding container 26d08c8357b740756ccc6a98a8ada865802b68bd21fb705c143f8fb29ff9d1af: Status 404 returned error can't find the container with id 26d08c8357b740756ccc6a98a8ada865802b68bd21fb705c143f8fb29ff9d1af
Sep 06 19:52:12.181339 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.181305    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:52:12.181454 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.181385    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:52:12.181454 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.181429    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:52:12.253273 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.253236    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h8xk\" (UniqueName: \"kubernetes.io/projected/d71087d1-64c2-4d4e-80ea-33081476865a-kube-api-access-6h8xk\") pod \"mt-curl-7639-b308da\" (UID: \"d71087d1-64c2-4d4e-80ea-33081476865a\") " pod="opendatahub/mt-curl-7639-b308da"
Sep 06 19:52:12.260796 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.260769    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h8xk\" (UniqueName: \"kubernetes.io/projected/d71087d1-64c2-4d4e-80ea-33081476865a-kube-api-access-6h8xk\") pod \"mt-curl-7639-b308da\" (UID: \"d71087d1-64c2-4d4e-80ea-33081476865a\") " pod="opendatahub/mt-curl-7639-b308da"
Sep 06 19:52:12.354258 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.354216    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7639-b308da"
Sep 06 19:52:12.626902 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.626862    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7639-b308da"]
Sep 06 19:52:12.629990 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:52:12.629939    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd71087d1_64c2_4d4e_80ea_33081476865a.slice/crio-fa061932c14c39a5b8a49101ee332ba3362be2ed09dd045705fd4008d2afd387 WatchSource:0}: Error finding container fa061932c14c39a5b8a49101ee332ba3362be2ed09dd045705fd4008d2afd387: Status 404 returned error can't find the container with id fa061932c14c39a5b8a49101ee332ba3362be2ed09dd045705fd4008d2afd387
Sep 06 19:52:12.750902 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.750862    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq" event={"ID":"ce637caa-427f-47a6-af77-6efff8fe023e","Type":"ContainerStarted","Data":"62aa54ecf5895f0af3994c3c9d1ece91b5fa0fee04bba56ea4c2dc919fee8a7f"}
Sep 06 19:52:12.751382 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.750918    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq" event={"ID":"ce637caa-427f-47a6-af77-6efff8fe023e","Type":"ContainerStarted","Data":"26d08c8357b740756ccc6a98a8ada865802b68bd21fb705c143f8fb29ff9d1af"}
Sep 06 19:52:12.752260 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.752241    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7639-fd4a6a"
Sep 06 19:52:12.752372 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.752247    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fe9089c658fb70d0831a7c9919d5e1ef074c88377294bcca194ae4614e4d641"
Sep 06 19:52:12.753531 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.753508    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7639-b308da" event={"ID":"d71087d1-64c2-4d4e-80ea-33081476865a","Type":"ContainerStarted","Data":"fa061932c14c39a5b8a49101ee332ba3362be2ed09dd045705fd4008d2afd387"}
Sep 06 19:52:12.788633 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.788577    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq" podStartSLOduration=1.788560879 podStartE2EDuration="1.788560879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:52:12.786138405 +0000 UTC m=+1496.356436752" watchObservedRunningTime="2026-09-06 19:52:12.788560879 +0000 UTC m=+1496.358859225"
Sep 06 19:52:12.788807 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:52:12.788603    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7639-fd4a6a\" is forbidden: User \"system:node:ip-10-0-141-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-42.ec2.internal' and this object" podUID="042955d3-46d1-4fd5-bfd0-cd5056a47fa2" pod="opendatahub/mt-curl-7639-fd4a6a"
Sep 06 19:52:12.904369 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.904311    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:12.919235 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:12.919172    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="042955d3-46d1-4fd5-bfd0-cd5056a47fa2" path="/var/lib/kubelet/pods/042955d3-46d1-4fd5-bfd0-cd5056a47fa2/volumes"
Sep 06 19:52:13.762561 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:13.762524    2571 generic.go:356] "Generic (PLEG): container finished" podID="d71087d1-64c2-4d4e-80ea-33081476865a" containerID="2615a4dd58fa758c4ac9e906368f956be019550aaebae85ce846d2f9663fffcd" exitCode=0
Sep 06 19:52:13.763013 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:13.762613    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7639-b308da" event={"ID":"d71087d1-64c2-4d4e-80ea-33081476865a","Type":"ContainerDied","Data":"2615a4dd58fa758c4ac9e906368f956be019550aaebae85ce846d2f9663fffcd"}
Sep 06 19:52:13.910572 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:13.910545    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:14.768180 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:14.768144    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:14.769213 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:14.769190    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:14.916120 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:14.916092    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7639-b308da"
Sep 06 19:52:14.937173 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:14.937141    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7639-b308da_d71087d1-64c2-4d4e-80ea-33081476865a/mt-curl-7639-b308da/0.log"
Sep 06 19:52:14.956714 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:14.956679    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7639-b308da"]
Sep 06 19:52:14.961297 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:14.961266    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7639-b308da"]
Sep 06 19:52:15.084692 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.084654    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h8xk\" (UniqueName: \"kubernetes.io/projected/d71087d1-64c2-4d4e-80ea-33081476865a-kube-api-access-6h8xk\") pod \"d71087d1-64c2-4d4e-80ea-33081476865a\" (UID: \"d71087d1-64c2-4d4e-80ea-33081476865a\") "
Sep 06 19:52:15.086820 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.086787    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71087d1-64c2-4d4e-80ea-33081476865a-kube-api-access-6h8xk" (OuterVolumeSpecName: "kube-api-access-6h8xk") pod "d71087d1-64c2-4d4e-80ea-33081476865a" (UID: "d71087d1-64c2-4d4e-80ea-33081476865a"). InnerVolumeSpecName "kube-api-access-6h8xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:52:15.185489 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.185443    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h8xk\" (UniqueName: \"kubernetes.io/projected/d71087d1-64c2-4d4e-80ea-33081476865a-kube-api-access-6h8xk\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:52:15.250719 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.250682    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:52:15.734898 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.734866    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"]
Sep 06 19:52:15.735319 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.735305    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d71087d1-64c2-4d4e-80ea-33081476865a" containerName="mt-curl-7639-b308da"
Sep 06 19:52:15.735393 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.735320    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d71087d1-64c2-4d4e-80ea-33081476865a" containerName="mt-curl-7639-b308da"
Sep 06 19:52:15.735393 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.735385    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d71087d1-64c2-4d4e-80ea-33081476865a" containerName="mt-curl-7639-b308da"
Sep 06 19:52:15.738766 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.738740    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:15.741198 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.741176    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cf16ea-serving-cert\""
Sep 06 19:52:15.750631 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.750599    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"]
Sep 06 19:52:15.774626 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.774596    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7639-b308da"
Sep 06 19:52:15.775085 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.774643    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa061932c14c39a5b8a49101ee332ba3362be2ed09dd045705fd4008d2afd387"
Sep 06 19:52:15.784677 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:52:15.784650    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7639-b308da\" is forbidden: User \"system:node:ip-10-0-141-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-42.ec2.internal' and this object" podUID="d71087d1-64c2-4d4e-80ea-33081476865a" pod="opendatahub/mt-curl-7639-b308da"
Sep 06 19:52:15.891811 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.891765    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whcqf\" (UniqueName: \"kubernetes.io/projected/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-kube-api-access-whcqf\") pod \"maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:15.892130 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.891975    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-metrics-certs\") pod \"maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:15.892130 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.892015    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-openshift-service-ca\") pod \"maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:15.892130 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.892115    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-maas-api-tls\") pod \"maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:15.992873 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.992782    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whcqf\" (UniqueName: \"kubernetes.io/projected/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-kube-api-access-whcqf\") pod \"maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:15.993057 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.992902    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-metrics-certs\") pod \"maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:15.993057 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.992930    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-openshift-service-ca\") pod \"maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:15.993057 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.992996    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-maas-api-tls\") pod \"maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:15.993217 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:52:15.993109    2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-serving-cert: secret "maas-api-e2e-api-a-cf16ea-serving-cert" not found
Sep 06 19:52:15.993217 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:52:15.993170    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-maas-api-tls podName:618a6f65-2d10-4cc6-8fe3-ec941aff03d1 nodeName:}" failed. No retries permitted until 2026-09-06 19:52:16.493150841 +0000 UTC m=+1500.063449171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-maas-api-tls") pod "maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t" (UID: "618a6f65-2d10-4cc6-8fe3-ec941aff03d1") : secret "maas-api-e2e-api-a-cf16ea-serving-cert" not found
Sep 06 19:52:15.993701 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.993673    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-openshift-service-ca\") pod \"maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:15.995475 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:15.995452    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-metrics-certs\") pod \"maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:16.000990 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:16.000949    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whcqf\" (UniqueName: \"kubernetes.io/projected/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-kube-api-access-whcqf\") pod \"maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:16.499138 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:16.499099    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-maas-api-tls\") pod \"maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:16.501602 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:16.501577    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-maas-api-tls\") pod \"maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:16.650726 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:16.650689    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:16.916885 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:52:16.916849    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7639-b308da\" is forbidden: User \"system:node:ip-10-0-141-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-42.ec2.internal' and this object" podUID="d71087d1-64c2-4d4e-80ea-33081476865a" pod="opendatahub/mt-curl-7639-b308da"
Sep 06 19:52:16.918932 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:16.918899    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d71087d1-64c2-4d4e-80ea-33081476865a" path="/var/lib/kubelet/pods/d71087d1-64c2-4d4e-80ea-33081476865a/volumes"
Sep 06 19:52:16.924524 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:16.924502    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"]
Sep 06 19:52:16.926537 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:52:16.926514    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618a6f65_2d10_4cc6_8fe3_ec941aff03d1.slice/crio-da3357703fed15101f053d054b503918129e1a70cef4f7955c718a4cba13d332 WatchSource:0}: Error finding container da3357703fed15101f053d054b503918129e1a70cef4f7955c718a4cba13d332: Status 404 returned error can't find the container with id da3357703fed15101f053d054b503918129e1a70cef4f7955c718a4cba13d332
Sep 06 19:52:16.993286 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:16.993258    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 19:52:16.995797 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:16.995770    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 19:52:17.184507 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:17.184421    2571 scope.go:117] "RemoveContainer" containerID="c0dfff963c2e4078f59763df50a0c1a04b0c9b9f6ad52cfe3ffcf256e4dd042e"
Sep 06 19:52:17.786164 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:17.786082    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t" event={"ID":"618a6f65-2d10-4cc6-8fe3-ec941aff03d1","Type":"ContainerStarted","Data":"a8b6be9df025df58e348fc304ad2b08d37305dcdc07059d1dd6b7c9cc537bc0a"}
Sep 06 19:52:17.786164 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:17.786160    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t" event={"ID":"618a6f65-2d10-4cc6-8fe3-ec941aff03d1","Type":"ContainerStarted","Data":"da3357703fed15101f053d054b503918129e1a70cef4f7955c718a4cba13d332"}
Sep 06 19:52:17.786352 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:17.786183    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:19.680711 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:19.680565    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t" podStartSLOduration=4.207688007 podStartE2EDuration="4.680541856s" totalImagesPullingTime="472.853849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:52:17.803175537 +0000 UTC m=+1501.373473885" watchObservedRunningTime="2026-09-06 19:52:19.680541856 +0000 UTC m=+1503.250840203"
Sep 06 19:52:19.682545 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:19.682520    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"]
Sep 06 19:52:23.797605 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:23.797572    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:52:27.948777 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:27.948727    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"]
Sep 06 19:52:27.968387 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:27.968344    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"]
Sep 06 19:52:27.968572 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:27.968456    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:27.972045 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:27.972014    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b571c4\"/\"kube-root-ca.crt\""
Sep 06 19:52:27.972299 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:27.972282    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b571c4\"/\"default-dockercfg-7dl84\""
Sep 06 19:52:27.972450 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:27.972424    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b571c4\"/\"openshift-service-ca.crt\""
Sep 06 19:52:27.972846 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:27.972827    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b571c4\"/\"rate-test-model-b571c4-kserve-self-signed-certs\""
Sep 06 19:52:28.010954 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.010913    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-dshm\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.010954 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.010985    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-model-cache\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.011216 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.011014    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l795h\" (UniqueName: \"kubernetes.io/projected/80e927b7-0065-4f5b-8299-a2e37dff3ccd-kube-api-access-l795h\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.011216 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.011082    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e927b7-0065-4f5b-8299-a2e37dff3ccd-tls-certs\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.011335 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.011236    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-home\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.011335 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.011273    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-tmp-dir\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.112278 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.112236    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-dshm\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.112278 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.112287    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-model-cache\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.112551 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.112316    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l795h\" (UniqueName: \"kubernetes.io/projected/80e927b7-0065-4f5b-8299-a2e37dff3ccd-kube-api-access-l795h\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.112551 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.112388    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e927b7-0065-4f5b-8299-a2e37dff3ccd-tls-certs\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.112551 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.112522    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-home\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.112712 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.112555    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-tmp-dir\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.113304 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.113175    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-home\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.113304 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.113255    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-tmp-dir\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.113304 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.113292    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-model-cache\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.115894 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.115856    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-dshm\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.116486 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.116457    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e927b7-0065-4f5b-8299-a2e37dff3ccd-tls-certs\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.119926 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.119902    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l795h\" (UniqueName: \"kubernetes.io/projected/80e927b7-0065-4f5b-8299-a2e37dff3ccd-kube-api-access-l795h\") pod \"rate-test-model-b571c4-kserve-7d545f984c-drcxg\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") " pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.279883 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.279775    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:28.564214 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.564186    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"]
Sep 06 19:52:28.566685 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:52:28.566657    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80e927b7_0065_4f5b_8299_a2e37dff3ccd.slice/crio-bf28bb0ea6a62b87dc3b4f99ed66aefd78b2a4efae4a6544e565823f6de92fec WatchSource:0}: Error finding container bf28bb0ea6a62b87dc3b4f99ed66aefd78b2a4efae4a6544e565823f6de92fec: Status 404 returned error can't find the container with id bf28bb0ea6a62b87dc3b4f99ed66aefd78b2a4efae4a6544e565823f6de92fec
Sep 06 19:52:28.851931 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.851818    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg" event={"ID":"80e927b7-0065-4f5b-8299-a2e37dff3ccd","Type":"ContainerStarted","Data":"3ea9be6c609fca649b6f70ab11e3bec6d9175dcc70f086817d86109947b231fb"}
Sep 06 19:52:28.852418 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.852391    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg" event={"ID":"80e927b7-0065-4f5b-8299-a2e37dff3ccd","Type":"ContainerStarted","Data":"bf28bb0ea6a62b87dc3b4f99ed66aefd78b2a4efae4a6544e565823f6de92fec"}
Sep 06 19:52:28.867694 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:28.867635    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg" podStartSLOduration=1.867618271 podStartE2EDuration="1.867618271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:52:28.866909589 +0000 UTC m=+1512.437207937" watchObservedRunningTime="2026-09-06 19:52:28.867618271 +0000 UTC m=+1512.437916617"
Sep 06 19:52:30.942269 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:30.942220    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:52:34.702857 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:34.702793    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn" podUID="8e86f22f-7bfd-42c9-a713-d09c80183cd1" containerName="main" containerID="cri-o://747f51563c62c6206ae9bba2bdfdb5d0f17eec3cabd865aa767da79bc9dac9af" gracePeriod=45
Sep 06 19:52:34.896062 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:34.896024    2571 generic.go:356] "Generic (PLEG): container finished" podID="8e86f22f-7bfd-42c9-a713-d09c80183cd1" containerID="747f51563c62c6206ae9bba2bdfdb5d0f17eec3cabd865aa767da79bc9dac9af" exitCode=0
Sep 06 19:52:34.896228 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:34.896067    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn" event={"ID":"8e86f22f-7bfd-42c9-a713-d09c80183cd1","Type":"ContainerDied","Data":"747f51563c62c6206ae9bba2bdfdb5d0f17eec3cabd865aa767da79bc9dac9af"}
Sep 06 19:52:34.898269 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:34.898240    2571 generic.go:356] "Generic (PLEG): container finished" podID="ce637caa-427f-47a6-af77-6efff8fe023e" containerID="62aa54ecf5895f0af3994c3c9d1ece91b5fa0fee04bba56ea4c2dc919fee8a7f" exitCode=0
Sep 06 19:52:34.898428 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:34.898302    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq" event={"ID":"ce637caa-427f-47a6-af77-6efff8fe023e","Type":"ContainerDied","Data":"62aa54ecf5895f0af3994c3c9d1ece91b5fa0fee04bba56ea4c2dc919fee8a7f"}
Sep 06 19:52:34.898772 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:34.898755    2571 scope.go:117] "RemoveContainer" containerID="62aa54ecf5895f0af3994c3c9d1ece91b5fa0fee04bba56ea4c2dc919fee8a7f"
Sep 06 19:52:34.900132 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:34.900102    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:52:34.900210 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:34.900180    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:52:34.900250 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:34.900214    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:52:34.980007 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:34.979982    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:52:35.091727 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.091679    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e86f22f-7bfd-42c9-a713-d09c80183cd1-tls-certs\") pod \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") "
Sep 06 19:52:35.091907 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.091776    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khq26\" (UniqueName: \"kubernetes.io/projected/8e86f22f-7bfd-42c9-a713-d09c80183cd1-kube-api-access-khq26\") pod \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") "
Sep 06 19:52:35.091907 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.091818    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-dshm\") pod \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") "
Sep 06 19:52:35.091907 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.091900    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-tmp-dir\") pod \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") "
Sep 06 19:52:35.092044 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.092000    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-model-cache\") pod \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") "
Sep 06 19:52:35.092093 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.092046    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-home\") pod \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\" (UID: \"8e86f22f-7bfd-42c9-a713-d09c80183cd1\") "
Sep 06 19:52:35.092323 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.092273    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e86f22f-7bfd-42c9-a713-d09c80183cd1" (UID: "8e86f22f-7bfd-42c9-a713-d09c80183cd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:52:35.092424 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.092320    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e86f22f-7bfd-42c9-a713-d09c80183cd1" (UID: "8e86f22f-7bfd-42c9-a713-d09c80183cd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:52:35.092483 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.092442    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-tmp-dir\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:52:35.092483 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.092463    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-model-cache\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:52:35.092581 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.092523    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-home" (OuterVolumeSpecName: "home") pod "8e86f22f-7bfd-42c9-a713-d09c80183cd1" (UID: "8e86f22f-7bfd-42c9-a713-d09c80183cd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:52:35.094161 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.094135    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e86f22f-7bfd-42c9-a713-d09c80183cd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e86f22f-7bfd-42c9-a713-d09c80183cd1" (UID: "8e86f22f-7bfd-42c9-a713-d09c80183cd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:52:35.094351 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.094327    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-dshm" (OuterVolumeSpecName: "dshm") pod "8e86f22f-7bfd-42c9-a713-d09c80183cd1" (UID: "8e86f22f-7bfd-42c9-a713-d09c80183cd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:52:35.094409 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.094372    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e86f22f-7bfd-42c9-a713-d09c80183cd1-kube-api-access-khq26" (OuterVolumeSpecName: "kube-api-access-khq26") pod "8e86f22f-7bfd-42c9-a713-d09c80183cd1" (UID: "8e86f22f-7bfd-42c9-a713-d09c80183cd1"). InnerVolumeSpecName "kube-api-access-khq26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:52:35.193667 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.193626    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-home\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:52:35.193667 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.193667    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e86f22f-7bfd-42c9-a713-d09c80183cd1-tls-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:52:35.193876 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.193683    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khq26\" (UniqueName: \"kubernetes.io/projected/8e86f22f-7bfd-42c9-a713-d09c80183cd1-kube-api-access-khq26\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:52:35.193876 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.193698    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e86f22f-7bfd-42c9-a713-d09c80183cd1-dshm\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:52:35.904539 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.904504    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"
Sep 06 19:52:35.904539 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.904525    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn" event={"ID":"8e86f22f-7bfd-42c9-a713-d09c80183cd1","Type":"ContainerDied","Data":"f1c5060777a7e92e466bc73ea0bab2fbf0a30d4f3fb6fe336a2f6b45f7c198a8"}
Sep 06 19:52:35.905097 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.904593    2571 scope.go:117] "RemoveContainer" containerID="747f51563c62c6206ae9bba2bdfdb5d0f17eec3cabd865aa767da79bc9dac9af"
Sep 06 19:52:35.906787 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.906759    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq" event={"ID":"ce637caa-427f-47a6-af77-6efff8fe023e","Type":"ContainerStarted","Data":"3cde58c6cdc9e6974379ee89d2a5ab2890be3a735f30cc76d0a0c30ddf7d8faf"}
Sep 06 19:52:35.937632 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.937595    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"]
Sep 06 19:52:35.940905 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:35.940863    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/sub-test-model-712616-kserve-fb6d8cbb9-hknxn"]
Sep 06 19:52:36.904358 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:36.904317    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:36.904554 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:36.904532    2571 patch_prober.go:28] interesting pod/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 06 19:52:36.904606 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:36.904580    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq" podUID="ce637caa-427f-47a6-af77-6efff8fe023e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 06 19:52:36.919456 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:36.919422    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e86f22f-7bfd-42c9-a713-d09c80183cd1" path="/var/lib/kubelet/pods/8e86f22f-7bfd-42c9-a713-d09c80183cd1/volumes"
Sep 06 19:52:37.904535 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:37.904499    2571 patch_prober.go:28] interesting pod/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 06 19:52:37.904730 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:37.904554    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq" podUID="ce637caa-427f-47a6-af77-6efff8fe023e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 06 19:52:38.280103 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:38.280008    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:38.280103 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:38.280093    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:38.293085 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:38.293050    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:38.904801 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:38.904715    2571 patch_prober.go:28] interesting pod/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 06 19:52:38.904801 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:38.904771    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq" podUID="ce637caa-427f-47a6-af77-6efff8fe023e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 06 19:52:38.959931 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:38.959893    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:52:39.905209 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:39.905165    2571 patch_prober.go:28] interesting pod/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 06 19:52:39.905611 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:39.905246    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq" podUID="ce637caa-427f-47a6-af77-6efff8fe023e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 06 19:52:40.281865 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.281771    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"]
Sep 06 19:52:40.282256 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.282241    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e86f22f-7bfd-42c9-a713-d09c80183cd1" containerName="main"
Sep 06 19:52:40.282332 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.282258    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e86f22f-7bfd-42c9-a713-d09c80183cd1" containerName="main"
Sep 06 19:52:40.282390 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.282341    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e86f22f-7bfd-42c9-a713-d09c80183cd1" containerName="main"
Sep 06 19:52:40.287026 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.287005    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.289864 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.289836    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d50bae-openshift-default-dockercfg-nzk92\""
Sep 06 19:52:40.298794 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.298767    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"]
Sep 06 19:52:40.352984 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.350504    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-data\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.352984 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.350590    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-workload-certs\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.352984 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.350652    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-workload-socket\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.352984 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.350677    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-credential-socket\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.352984 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.350715    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-envoy\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.352984 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.350739    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istiod-ca-cert\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.352984 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.350771    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bchhg\" (UniqueName: \"kubernetes.io/projected/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-kube-api-access-bchhg\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.352984 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.350818    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-token\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.352984 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.350845    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-podinfo\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.451846 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.451802    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-workload-certs\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.452241 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.452219    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-workload-socket\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.452707 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.452689    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-credential-socket\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.453184 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.453164    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-envoy\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.453351 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.453336    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istiod-ca-cert\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.453472 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.453456    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bchhg\" (UniqueName: \"kubernetes.io/projected/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-kube-api-access-bchhg\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.453612 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.453597    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-token\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.454272 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.454252    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-podinfo\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.454461 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.454443    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-data\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.454623 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.452648    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-workload-socket\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.454733 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.454308    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istiod-ca-cert\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.454826 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.452495    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-workload-certs\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.455319 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.453095    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-credential-socket\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.455439 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.454930    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-data\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.457382 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.457361    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-podinfo\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.457797 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.457779    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-envoy\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.463548 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.463524    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-token\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.464207 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.464181    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bchhg\" (UniqueName: \"kubernetes.io/projected/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-kube-api-access-bchhg\") pod \"e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") " pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.600606 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.600563    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:40.899189 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.899161    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"]
Sep 06 19:52:40.901299 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:52:40.901252    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558eb546_2a5e_4d73_ac28_0c3a07a5bd4e.slice/crio-88e450938c0798d203c86501890ac917f243761ab6dfd0e2007e1a61a6cea565 WatchSource:0}: Error finding container 88e450938c0798d203c86501890ac917f243761ab6dfd0e2007e1a61a6cea565: Status 404 returned error can't find the container with id 88e450938c0798d203c86501890ac917f243761ab6dfd0e2007e1a61a6cea565
Sep 06 19:52:40.903910 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.903542    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:52:40.903910 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.903609    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:52:40.903910 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.903637    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:52:40.904196 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.904173    2571 patch_prober.go:28] interesting pod/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 06 19:52:40.904247 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.904208    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq" podUID="ce637caa-427f-47a6-af77-6efff8fe023e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 06 19:52:40.948252 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:40.948211    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw" event={"ID":"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e","Type":"ContainerStarted","Data":"88e450938c0798d203c86501890ac917f243761ab6dfd0e2007e1a61a6cea565"}
Sep 06 19:52:41.908567 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:41.908532    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:41.908855 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:41.908837    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:41.909654 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:41.909632    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:52:41.957166 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:41.957125    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw" event={"ID":"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e","Type":"ContainerStarted","Data":"552b430995edf0fc70b609ba95f415a40f06b90f43c0c1677192e502a67d3d1e"}
Sep 06 19:52:41.977423 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:41.977373    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw" podStartSLOduration=1.9773570409999999 podStartE2EDuration="1.977357041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:52:41.976080267 +0000 UTC m=+1525.546378613" watchObservedRunningTime="2026-09-06 19:52:41.977357041 +0000 UTC m=+1525.547655386"
Sep 06 19:52:42.601099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:42.601065    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:42.606234 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:42.606206    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:42.963696 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:42.963609    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:42.968141 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:42.968108    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:52:52.448529 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:52.448489    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:52:59.737564 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.737529    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"]
Sep 06 19:52:59.740792 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.740769    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.744256 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.744229    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-712616\"/\"kube-root-ca.crt\""
Sep 06 19:52:59.744495 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.744292    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-712616\"/\"rate-test-model-712616-kserve-self-signed-certs\""
Sep 06 19:52:59.744495 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.744292    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-712616\"/\"default-dockercfg-mjc8d\""
Sep 06 19:52:59.744693 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.744347    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-712616\"/\"openshift-service-ca.crt\""
Sep 06 19:52:59.749703 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.749673    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"]
Sep 06 19:52:59.866638 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.866606    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcfgn\" (UniqueName: \"kubernetes.io/projected/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-kube-api-access-jcfgn\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.866789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.866648    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-home\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.866789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.866757    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-tls-certs\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.866789 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.866786    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-dshm\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.866923 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.866803    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-tmp-dir\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.866983 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.866939    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-model-cache\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.967580 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.967537    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-model-cache\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.967580 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.967588    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcfgn\" (UniqueName: \"kubernetes.io/projected/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-kube-api-access-jcfgn\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.967837 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.967625    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-home\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.967837 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.967714    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-tls-certs\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.967837 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.967738    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-dshm\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.967837 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.967756    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-tmp-dir\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.968151 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.968067    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-model-cache\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.968151 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.968134    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-home\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.968265 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.968224    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-tmp-dir\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.970146 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.970123    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-dshm\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.970405 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.970382    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-tls-certs\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:52:59.974794 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:52:59.974772    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcfgn\" (UniqueName: \"kubernetes.io/projected/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-kube-api-access-jcfgn\") pod \"rate-test-model-712616-kserve-65d4dff6c9-gz8wq\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") " pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:53:00.054955 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:00.054876    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:53:00.338631 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:00.336639    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"]
Sep 06 19:53:01.066436 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:01.066398    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq" event={"ID":"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c","Type":"ContainerStarted","Data":"335c4742ee7af67eacd81003f66f226a040acbcc324702e790f2ef82dcb4dda1"}
Sep 06 19:53:01.066436 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:01.066440    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq" event={"ID":"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c","Type":"ContainerStarted","Data":"3c93d39cf0365ce346d6a2352642cb267db50711704590a4338ab1cefb2259d2"}
Sep 06 19:53:01.082947 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:01.082892    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq" podStartSLOduration=2.082877451 podStartE2EDuration="2.082877451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:53:01.080824169 +0000 UTC m=+1544.651122516" watchObservedRunningTime="2026-09-06 19:53:01.082877451 +0000 UTC m=+1544.653175797"
Sep 06 19:53:03.080641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:03.080609    2571 generic.go:356] "Generic (PLEG): container finished" podID="558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" containerID="552b430995edf0fc70b609ba95f415a40f06b90f43c0c1677192e502a67d3d1e" exitCode=0
Sep 06 19:53:03.081087 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:03.080687    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw" event={"ID":"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e","Type":"ContainerDied","Data":"552b430995edf0fc70b609ba95f415a40f06b90f43c0c1677192e502a67d3d1e"}
Sep 06 19:53:03.081143 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:03.081129    2571 scope.go:117] "RemoveContainer" containerID="552b430995edf0fc70b609ba95f415a40f06b90f43c0c1677192e502a67d3d1e"
Sep 06 19:53:03.082348 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:03.082294    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:53:03.082409 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:03.082364    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:53:03.082445 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:03.082417    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:53:04.088208 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:04.088177    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw" event={"ID":"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e","Type":"ContainerStarted","Data":"a7b29b922cb3bd015ed164a55a0b43ff5186efcf02d3882f609c8f950bf32593"}
Sep 06 19:53:04.600788 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:04.600752    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:53:04.601014 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:04.600992    2571 patch_prober.go:28] interesting pod/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 06 19:53:04.601063 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:04.601040    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw" podUID="558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 06 19:53:05.601459 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:05.601419    2571 patch_prober.go:28] interesting pod/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 06 19:53:05.601892 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:05.601492    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw" podUID="558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 06 19:53:06.245912 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:06.245865    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:53:06.602020 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:06.601976    2571 patch_prober.go:28] interesting pod/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 06 19:53:06.602542 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:06.602048    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw" podUID="558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 06 19:53:07.602025 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:07.601983    2571 patch_prober.go:28] interesting pod/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 06 19:53:07.602207 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:07.602048    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw" podUID="558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 06 19:53:08.601834 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:08.601789    2571 patch_prober.go:28] interesting pod/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Sep 06 19:53:08.602046 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:08.601851    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw" podUID="558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Sep 06 19:53:09.605809 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:09.605775    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:53:09.606288 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:09.606078    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:53:09.606900 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:09.606884    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:53:10.055073 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:10.054990    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:53:10.055073 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:10.055047    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:53:10.067631 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:10.067607    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:53:10.130776 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:10.130749    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:53:27.038934 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.038900    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"]
Sep 06 19:53:27.039374 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.039175    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t" podUID="618a6f65-2d10-4cc6-8fe3-ec941aff03d1" containerName="maas-api" containerID="cri-o://a8b6be9df025df58e348fc304ad2b08d37305dcdc07059d1dd6b7c9cc537bc0a" gracePeriod=30
Sep 06 19:53:27.220064 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.220029    2571 generic.go:356] "Generic (PLEG): container finished" podID="618a6f65-2d10-4cc6-8fe3-ec941aff03d1" containerID="a8b6be9df025df58e348fc304ad2b08d37305dcdc07059d1dd6b7c9cc537bc0a" exitCode=0
Sep 06 19:53:27.220221 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.220103    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t" event={"ID":"618a6f65-2d10-4cc6-8fe3-ec941aff03d1","Type":"ContainerDied","Data":"a8b6be9df025df58e348fc304ad2b08d37305dcdc07059d1dd6b7c9cc537bc0a"}
Sep 06 19:53:27.284616 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.284589    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:53:27.356267 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.356237    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-metrics-certs\") pod \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") "
Sep 06 19:53:27.356498 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.356293    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-maas-api-tls\") pod \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") "
Sep 06 19:53:27.356498 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.356352    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whcqf\" (UniqueName: \"kubernetes.io/projected/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-kube-api-access-whcqf\") pod \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") "
Sep 06 19:53:27.356498 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.356457    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-openshift-service-ca\") pod \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\" (UID: \"618a6f65-2d10-4cc6-8fe3-ec941aff03d1\") "
Sep 06 19:53:27.356863 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.356836    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "618a6f65-2d10-4cc6-8fe3-ec941aff03d1" (UID: "618a6f65-2d10-4cc6-8fe3-ec941aff03d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:53:27.358688 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.358662    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "618a6f65-2d10-4cc6-8fe3-ec941aff03d1" (UID: "618a6f65-2d10-4cc6-8fe3-ec941aff03d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:53:27.358912 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.358881    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "618a6f65-2d10-4cc6-8fe3-ec941aff03d1" (UID: "618a6f65-2d10-4cc6-8fe3-ec941aff03d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:53:27.359300 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.359273    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-kube-api-access-whcqf" (OuterVolumeSpecName: "kube-api-access-whcqf") pod "618a6f65-2d10-4cc6-8fe3-ec941aff03d1" (UID: "618a6f65-2d10-4cc6-8fe3-ec941aff03d1"). InnerVolumeSpecName "kube-api-access-whcqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:53:27.458406 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.458364    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whcqf\" (UniqueName: \"kubernetes.io/projected/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-kube-api-access-whcqf\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:53:27.458406 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.458402    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-openshift-service-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:53:27.458406 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.458415    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-metrics-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:53:27.458658 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:27.458427    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/618a6f65-2d10-4cc6-8fe3-ec941aff03d1-maas-api-tls\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:53:28.227187 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:28.227157    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"
Sep 06 19:53:28.227619 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:28.227198    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t" event={"ID":"618a6f65-2d10-4cc6-8fe3-ec941aff03d1","Type":"ContainerDied","Data":"da3357703fed15101f053d054b503918129e1a70cef4f7955c718a4cba13d332"}
Sep 06 19:53:28.227619 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:28.227261    2571 scope.go:117] "RemoveContainer" containerID="a8b6be9df025df58e348fc304ad2b08d37305dcdc07059d1dd6b7c9cc537bc0a"
Sep 06 19:53:28.249696 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:28.249668    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"]
Sep 06 19:53:28.253142 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:28.253115    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cf16ea-7478dfbc5c-fdh4t"]
Sep 06 19:53:28.643097 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:28.643065    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:53:28.920005 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:28.919912    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="618a6f65-2d10-4cc6-8fe3-ec941aff03d1" path="/var/lib/kubelet/pods/618a6f65-2d10-4cc6-8fe3-ec941aff03d1/volumes"
Sep 06 19:53:33.534303 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:33.534269    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"]
Sep 06 19:53:33.534842 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:33.534537    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq" podUID="ce637caa-427f-47a6-af77-6efff8fe023e" containerName="istio-proxy" containerID="cri-o://3cde58c6cdc9e6974379ee89d2a5ab2890be3a735f30cc76d0a0c30ddf7d8faf" gracePeriod=30
Sep 06 19:53:38.796852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.796818    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:53:38.879770 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.879732    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-workload-certs\") pod \"ce637caa-427f-47a6-af77-6efff8fe023e\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") "
Sep 06 19:53:38.880012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.879798    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce637caa-427f-47a6-af77-6efff8fe023e-istio-podinfo\") pod \"ce637caa-427f-47a6-af77-6efff8fe023e\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") "
Sep 06 19:53:38.880012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.879820    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-istio-envoy\") pod \"ce637caa-427f-47a6-af77-6efff8fe023e\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") "
Sep 06 19:53:38.880012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.879835    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce637caa-427f-47a6-af77-6efff8fe023e-istio-token\") pod \"ce637caa-427f-47a6-af77-6efff8fe023e\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") "
Sep 06 19:53:38.880012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.879866    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce637caa-427f-47a6-af77-6efff8fe023e-istiod-ca-cert\") pod \"ce637caa-427f-47a6-af77-6efff8fe023e\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") "
Sep 06 19:53:38.880012 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.879894    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-workload-socket\") pod \"ce637caa-427f-47a6-af77-6efff8fe023e\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") "
Sep 06 19:53:38.880373 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.880030    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-credential-socket\") pod \"ce637caa-427f-47a6-af77-6efff8fe023e\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") "
Sep 06 19:53:38.880373 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.880077    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-istio-data\") pod \"ce637caa-427f-47a6-af77-6efff8fe023e\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") "
Sep 06 19:53:38.880373 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.880074    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ce637caa-427f-47a6-af77-6efff8fe023e" (UID: "ce637caa-427f-47a6-af77-6efff8fe023e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:53:38.880373 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.880138    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7tr6\" (UniqueName: \"kubernetes.io/projected/ce637caa-427f-47a6-af77-6efff8fe023e-kube-api-access-f7tr6\") pod \"ce637caa-427f-47a6-af77-6efff8fe023e\" (UID: \"ce637caa-427f-47a6-af77-6efff8fe023e\") "
Sep 06 19:53:38.880373 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.880241    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ce637caa-427f-47a6-af77-6efff8fe023e" (UID: "ce637caa-427f-47a6-af77-6efff8fe023e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:53:38.880645 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.880401    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ce637caa-427f-47a6-af77-6efff8fe023e" (UID: "ce637caa-427f-47a6-af77-6efff8fe023e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:53:38.880645 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.880458    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-istio-data" (OuterVolumeSpecName: "istio-data") pod "ce637caa-427f-47a6-af77-6efff8fe023e" (UID: "ce637caa-427f-47a6-af77-6efff8fe023e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:53:38.880645 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.880496    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce637caa-427f-47a6-af77-6efff8fe023e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ce637caa-427f-47a6-af77-6efff8fe023e" (UID: "ce637caa-427f-47a6-af77-6efff8fe023e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:53:38.880645 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.880600    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce637caa-427f-47a6-af77-6efff8fe023e-istiod-ca-cert\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:53:38.880645 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.880618    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-workload-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:53:38.880645 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.880630    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-credential-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:53:38.880645 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.880644    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-istio-data\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:53:38.880903 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.880656    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-workload-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:53:38.882159 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.882135    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce637caa-427f-47a6-af77-6efff8fe023e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ce637caa-427f-47a6-af77-6efff8fe023e" (UID: "ce637caa-427f-47a6-af77-6efff8fe023e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 19:53:38.882388 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.882371    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce637caa-427f-47a6-af77-6efff8fe023e-kube-api-access-f7tr6" (OuterVolumeSpecName: "kube-api-access-f7tr6") pod "ce637caa-427f-47a6-af77-6efff8fe023e" (UID: "ce637caa-427f-47a6-af77-6efff8fe023e"). InnerVolumeSpecName "kube-api-access-f7tr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:53:38.882554 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.882531    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce637caa-427f-47a6-af77-6efff8fe023e-istio-token" (OuterVolumeSpecName: "istio-token") pod "ce637caa-427f-47a6-af77-6efff8fe023e" (UID: "ce637caa-427f-47a6-af77-6efff8fe023e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:53:38.882647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.882532    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ce637caa-427f-47a6-af77-6efff8fe023e" (UID: "ce637caa-427f-47a6-af77-6efff8fe023e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:53:38.981466 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.981428    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce637caa-427f-47a6-af77-6efff8fe023e-istio-podinfo\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:53:38.981466 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.981461    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce637caa-427f-47a6-af77-6efff8fe023e-istio-envoy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:53:38.981466 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.981469    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce637caa-427f-47a6-af77-6efff8fe023e-istio-token\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:53:38.981708 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:38.981478    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7tr6\" (UniqueName: \"kubernetes.io/projected/ce637caa-427f-47a6-af77-6efff8fe023e-kube-api-access-f7tr6\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:53:39.292211 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:39.292116    2571 generic.go:356] "Generic (PLEG): container finished" podID="ce637caa-427f-47a6-af77-6efff8fe023e" containerID="3cde58c6cdc9e6974379ee89d2a5ab2890be3a735f30cc76d0a0c30ddf7d8faf" exitCode=0
Sep 06 19:53:39.292211 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:39.292168    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"
Sep 06 19:53:39.292211 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:39.292196    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq" event={"ID":"ce637caa-427f-47a6-af77-6efff8fe023e","Type":"ContainerDied","Data":"3cde58c6cdc9e6974379ee89d2a5ab2890be3a735f30cc76d0a0c30ddf7d8faf"}
Sep 06 19:53:39.292457 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:39.292265    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq" event={"ID":"ce637caa-427f-47a6-af77-6efff8fe023e","Type":"ContainerDied","Data":"26d08c8357b740756ccc6a98a8ada865802b68bd21fb705c143f8fb29ff9d1af"}
Sep 06 19:53:39.292457 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:39.292291    2571 scope.go:117] "RemoveContainer" containerID="3cde58c6cdc9e6974379ee89d2a5ab2890be3a735f30cc76d0a0c30ddf7d8faf"
Sep 06 19:53:39.302282 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:39.302263    2571 scope.go:117] "RemoveContainer" containerID="62aa54ecf5895f0af3994c3c9d1ece91b5fa0fee04bba56ea4c2dc919fee8a7f"
Sep 06 19:53:39.309142 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:39.309108    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"]
Sep 06 19:53:39.311437 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:39.311415    2571 scope.go:117] "RemoveContainer" containerID="3cde58c6cdc9e6974379ee89d2a5ab2890be3a735f30cc76d0a0c30ddf7d8faf"
Sep 06 19:53:39.311866 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:53:39.311839    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cde58c6cdc9e6974379ee89d2a5ab2890be3a735f30cc76d0a0c30ddf7d8faf\": container with ID starting with 3cde58c6cdc9e6974379ee89d2a5ab2890be3a735f30cc76d0a0c30ddf7d8faf not found: ID does not exist" containerID="3cde58c6cdc9e6974379ee89d2a5ab2890be3a735f30cc76d0a0c30ddf7d8faf"
Sep 06 19:53:39.312009 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:39.311913    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cde58c6cdc9e6974379ee89d2a5ab2890be3a735f30cc76d0a0c30ddf7d8faf"} err="failed to get container status \"3cde58c6cdc9e6974379ee89d2a5ab2890be3a735f30cc76d0a0c30ddf7d8faf\": rpc error: code = NotFound desc = could not find container \"3cde58c6cdc9e6974379ee89d2a5ab2890be3a735f30cc76d0a0c30ddf7d8faf\": container with ID starting with 3cde58c6cdc9e6974379ee89d2a5ab2890be3a735f30cc76d0a0c30ddf7d8faf not found: ID does not exist"
Sep 06 19:53:39.312096 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:39.312010    2571 scope.go:117] "RemoveContainer" containerID="62aa54ecf5895f0af3994c3c9d1ece91b5fa0fee04bba56ea4c2dc919fee8a7f"
Sep 06 19:53:39.312579 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:53:39.312558    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62aa54ecf5895f0af3994c3c9d1ece91b5fa0fee04bba56ea4c2dc919fee8a7f\": container with ID starting with 62aa54ecf5895f0af3994c3c9d1ece91b5fa0fee04bba56ea4c2dc919fee8a7f not found: ID does not exist" containerID="62aa54ecf5895f0af3994c3c9d1ece91b5fa0fee04bba56ea4c2dc919fee8a7f"
Sep 06 19:53:39.312680 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:39.312583    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62aa54ecf5895f0af3994c3c9d1ece91b5fa0fee04bba56ea4c2dc919fee8a7f"} err="failed to get container status \"62aa54ecf5895f0af3994c3c9d1ece91b5fa0fee04bba56ea4c2dc919fee8a7f\": rpc error: code = NotFound desc = could not find container \"62aa54ecf5895f0af3994c3c9d1ece91b5fa0fee04bba56ea4c2dc919fee8a7f\": container with ID starting with 62aa54ecf5895f0af3994c3c9d1ece91b5fa0fee04bba56ea4c2dc919fee8a7f not found: ID does not exist"
Sep 06 19:53:39.314516 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:39.314497    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cf16ea-openshift-default-6655c5d468-tmpvq"]
Sep 06 19:53:40.918819 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:40.918785    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce637caa-427f-47a6-af77-6efff8fe023e" path="/var/lib/kubelet/pods/ce637caa-427f-47a6-af77-6efff8fe023e/volumes"
Sep 06 19:53:46.536922 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:46.536884    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:53:51.336320 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:51.336284    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"]
Sep 06 19:53:52.350051 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:52.350020    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"]
Sep 06 19:53:55.849243 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:55.849205    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"]
Sep 06 19:53:55.849727 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:53:55.849476    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw" podUID="558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" containerName="istio-proxy" containerID="cri-o://a7b29b922cb3bd015ed164a55a0b43ff5186efcf02d3882f609c8f950bf32593" gracePeriod=30
Sep 06 19:54:01.106378 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.106350    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:54:01.200226 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200141    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-envoy\") pod \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") "
Sep 06 19:54:01.200226 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200209    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-podinfo\") pod \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") "
Sep 06 19:54:01.200226 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200230    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-workload-certs\") pod \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") "
Sep 06 19:54:01.200526 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200251    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bchhg\" (UniqueName: \"kubernetes.io/projected/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-kube-api-access-bchhg\") pod \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") "
Sep 06 19:54:01.200526 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200298    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-token\") pod \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") "
Sep 06 19:54:01.200526 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200338    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-data\") pod \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") "
Sep 06 19:54:01.200526 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200362    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-workload-socket\") pod \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") "
Sep 06 19:54:01.200526 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200424    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-credential-socket\") pod \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") "
Sep 06 19:54:01.200526 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200449    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istiod-ca-cert\") pod \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\" (UID: \"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e\") "
Sep 06 19:54:01.200847 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200565    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" (UID: "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:01.200847 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200622    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-data" (OuterVolumeSpecName: "istio-data") pod "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" (UID: "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:01.200847 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200758    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" (UID: "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:01.200847 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200825    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-workload-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:01.200847 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200843    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-data\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:01.201138 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200856    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-workload-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:01.201138 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200932    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" (UID: "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:01.201138 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.200994    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" (UID: "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:54:01.202564 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.202538    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" (UID: "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 19:54:01.209901 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.202695    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-token" (OuterVolumeSpecName: "istio-token") pod "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" (UID: "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:54:01.209901 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.203091    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-kube-api-access-bchhg" (OuterVolumeSpecName: "kube-api-access-bchhg") pod "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" (UID: "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e"). InnerVolumeSpecName "kube-api-access-bchhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:54:01.209901 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.203097    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" (UID: "558eb546-2a5e-4d73-ac28-0c3a07a5bd4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:01.301618 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.301579    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-credential-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:01.301618 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.301612    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istiod-ca-cert\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:01.301618 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.301623    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-envoy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:01.301862 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.301631    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-podinfo\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:01.301862 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.301639    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bchhg\" (UniqueName: \"kubernetes.io/projected/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-kube-api-access-bchhg\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:01.301862 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.301648    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e-istio-token\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:01.416409 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.416375    2571 generic.go:356] "Generic (PLEG): container finished" podID="558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" containerID="a7b29b922cb3bd015ed164a55a0b43ff5186efcf02d3882f609c8f950bf32593" exitCode=0
Sep 06 19:54:01.416585 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.416432    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw" event={"ID":"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e","Type":"ContainerDied","Data":"a7b29b922cb3bd015ed164a55a0b43ff5186efcf02d3882f609c8f950bf32593"}
Sep 06 19:54:01.416585 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.416441    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"
Sep 06 19:54:01.416585 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.416498    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw" event={"ID":"558eb546-2a5e-4d73-ac28-0c3a07a5bd4e","Type":"ContainerDied","Data":"88e450938c0798d203c86501890ac917f243761ab6dfd0e2007e1a61a6cea565"}
Sep 06 19:54:01.416585 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.416521    2571 scope.go:117] "RemoveContainer" containerID="a7b29b922cb3bd015ed164a55a0b43ff5186efcf02d3882f609c8f950bf32593"
Sep 06 19:54:01.429754 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.429734    2571 scope.go:117] "RemoveContainer" containerID="552b430995edf0fc70b609ba95f415a40f06b90f43c0c1677192e502a67d3d1e"
Sep 06 19:54:01.439112 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.439085    2571 scope.go:117] "RemoveContainer" containerID="a7b29b922cb3bd015ed164a55a0b43ff5186efcf02d3882f609c8f950bf32593"
Sep 06 19:54:01.439494 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:54:01.439473    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b29b922cb3bd015ed164a55a0b43ff5186efcf02d3882f609c8f950bf32593\": container with ID starting with a7b29b922cb3bd015ed164a55a0b43ff5186efcf02d3882f609c8f950bf32593 not found: ID does not exist" containerID="a7b29b922cb3bd015ed164a55a0b43ff5186efcf02d3882f609c8f950bf32593"
Sep 06 19:54:01.439563 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.439503    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b29b922cb3bd015ed164a55a0b43ff5186efcf02d3882f609c8f950bf32593"} err="failed to get container status \"a7b29b922cb3bd015ed164a55a0b43ff5186efcf02d3882f609c8f950bf32593\": rpc error: code = NotFound desc = could not find container \"a7b29b922cb3bd015ed164a55a0b43ff5186efcf02d3882f609c8f950bf32593\": container with ID starting with a7b29b922cb3bd015ed164a55a0b43ff5186efcf02d3882f609c8f950bf32593 not found: ID does not exist"
Sep 06 19:54:01.439563 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.439524    2571 scope.go:117] "RemoveContainer" containerID="552b430995edf0fc70b609ba95f415a40f06b90f43c0c1677192e502a67d3d1e"
Sep 06 19:54:01.439813 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.439782    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"]
Sep 06 19:54:01.439877 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:54:01.439812    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552b430995edf0fc70b609ba95f415a40f06b90f43c0c1677192e502a67d3d1e\": container with ID starting with 552b430995edf0fc70b609ba95f415a40f06b90f43c0c1677192e502a67d3d1e not found: ID does not exist" containerID="552b430995edf0fc70b609ba95f415a40f06b90f43c0c1677192e502a67d3d1e"
Sep 06 19:54:01.439877 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.439827    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552b430995edf0fc70b609ba95f415a40f06b90f43c0c1677192e502a67d3d1e"} err="failed to get container status \"552b430995edf0fc70b609ba95f415a40f06b90f43c0c1677192e502a67d3d1e\": rpc error: code = NotFound desc = could not find container \"552b430995edf0fc70b609ba95f415a40f06b90f43c0c1677192e502a67d3d1e\": container with ID starting with 552b430995edf0fc70b609ba95f415a40f06b90f43c0c1677192e502a67d3d1e not found: ID does not exist"
Sep 06 19:54:01.443219 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:01.443193    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d50bae-openshift-default-7d4856664-cfkvw"]
Sep 06 19:54:02.919496 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:02.919459    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" path="/var/lib/kubelet/pods/558eb546-2a5e-4d73-ac28-0c3a07a5bd4e/volumes"
Sep 06 19:54:03.642809 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:03.642773    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:54:06.356112 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.356073    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg" podUID="80e927b7-0065-4f5b-8299-a2e37dff3ccd" containerName="main" containerID="cri-o://3ea9be6c609fca649b6f70ab11e3bec6d9175dcc70f086817d86109947b231fb" gracePeriod=45
Sep 06 19:54:06.609380 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.609320    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:54:06.654641 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.654602    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l795h\" (UniqueName: \"kubernetes.io/projected/80e927b7-0065-4f5b-8299-a2e37dff3ccd-kube-api-access-l795h\") pod \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") "
Sep 06 19:54:06.654852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.654657    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-model-cache\") pod \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") "
Sep 06 19:54:06.654852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.654755    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-home\") pod \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") "
Sep 06 19:54:06.654852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.654799    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e927b7-0065-4f5b-8299-a2e37dff3ccd-tls-certs\") pod \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") "
Sep 06 19:54:06.654852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.654835    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-dshm\") pod \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") "
Sep 06 19:54:06.655127 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.654883    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-tmp-dir\") pod \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\" (UID: \"80e927b7-0065-4f5b-8299-a2e37dff3ccd\") "
Sep 06 19:54:06.655127 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.655008    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-model-cache" (OuterVolumeSpecName: "model-cache") pod "80e927b7-0065-4f5b-8299-a2e37dff3ccd" (UID: "80e927b7-0065-4f5b-8299-a2e37dff3ccd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:06.655127 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.655055    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-home" (OuterVolumeSpecName: "home") pod "80e927b7-0065-4f5b-8299-a2e37dff3ccd" (UID: "80e927b7-0065-4f5b-8299-a2e37dff3ccd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:06.655283 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.655205    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80e927b7-0065-4f5b-8299-a2e37dff3ccd" (UID: "80e927b7-0065-4f5b-8299-a2e37dff3ccd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:06.655376 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.655356    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-tmp-dir\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:06.655431 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.655380    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-model-cache\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:06.655431 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.655395    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-home\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:06.657004 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.656972    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e927b7-0065-4f5b-8299-a2e37dff3ccd-kube-api-access-l795h" (OuterVolumeSpecName: "kube-api-access-l795h") pod "80e927b7-0065-4f5b-8299-a2e37dff3ccd" (UID: "80e927b7-0065-4f5b-8299-a2e37dff3ccd"). InnerVolumeSpecName "kube-api-access-l795h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:54:06.657120 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.657025    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-dshm" (OuterVolumeSpecName: "dshm") pod "80e927b7-0065-4f5b-8299-a2e37dff3ccd" (UID: "80e927b7-0065-4f5b-8299-a2e37dff3ccd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:06.657120 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.657041    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e927b7-0065-4f5b-8299-a2e37dff3ccd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80e927b7-0065-4f5b-8299-a2e37dff3ccd" (UID: "80e927b7-0065-4f5b-8299-a2e37dff3ccd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:54:06.756326 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.756289    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e927b7-0065-4f5b-8299-a2e37dff3ccd-tls-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:06.756326 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.756319    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e927b7-0065-4f5b-8299-a2e37dff3ccd-dshm\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:06.756326 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:06.756328    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l795h\" (UniqueName: \"kubernetes.io/projected/80e927b7-0065-4f5b-8299-a2e37dff3ccd-kube-api-access-l795h\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:07.370059 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.370024    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq" podUID="630cf613-76cb-44a7-94e7-5bb1d4d4dd0c" containerName="main" containerID="cri-o://335c4742ee7af67eacd81003f66f226a040acbcc324702e790f2ef82dcb4dda1" gracePeriod=45
Sep 06 19:54:07.456453 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.456418    2571 generic.go:356] "Generic (PLEG): container finished" podID="80e927b7-0065-4f5b-8299-a2e37dff3ccd" containerID="3ea9be6c609fca649b6f70ab11e3bec6d9175dcc70f086817d86109947b231fb" exitCode=0
Sep 06 19:54:07.456621 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.456460    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"
Sep 06 19:54:07.456621 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.456505    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg" event={"ID":"80e927b7-0065-4f5b-8299-a2e37dff3ccd","Type":"ContainerDied","Data":"3ea9be6c609fca649b6f70ab11e3bec6d9175dcc70f086817d86109947b231fb"}
Sep 06 19:54:07.456621 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.456544    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg" event={"ID":"80e927b7-0065-4f5b-8299-a2e37dff3ccd","Type":"ContainerDied","Data":"bf28bb0ea6a62b87dc3b4f99ed66aefd78b2a4efae4a6544e565823f6de92fec"}
Sep 06 19:54:07.456621 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.456561    2571 scope.go:117] "RemoveContainer" containerID="3ea9be6c609fca649b6f70ab11e3bec6d9175dcc70f086817d86109947b231fb"
Sep 06 19:54:07.473233 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.473200    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"]
Sep 06 19:54:07.477168 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.477137    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b571c4/rate-test-model-b571c4-kserve-7d545f984c-drcxg"]
Sep 06 19:54:07.482387 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.482367    2571 scope.go:117] "RemoveContainer" containerID="3ea9be6c609fca649b6f70ab11e3bec6d9175dcc70f086817d86109947b231fb"
Sep 06 19:54:07.482720 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:54:07.482695    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea9be6c609fca649b6f70ab11e3bec6d9175dcc70f086817d86109947b231fb\": container with ID starting with 3ea9be6c609fca649b6f70ab11e3bec6d9175dcc70f086817d86109947b231fb not found: ID does not exist" containerID="3ea9be6c609fca649b6f70ab11e3bec6d9175dcc70f086817d86109947b231fb"
Sep 06 19:54:07.482790 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.482735    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea9be6c609fca649b6f70ab11e3bec6d9175dcc70f086817d86109947b231fb"} err="failed to get container status \"3ea9be6c609fca649b6f70ab11e3bec6d9175dcc70f086817d86109947b231fb\": rpc error: code = NotFound desc = could not find container \"3ea9be6c609fca649b6f70ab11e3bec6d9175dcc70f086817d86109947b231fb\": container with ID starting with 3ea9be6c609fca649b6f70ab11e3bec6d9175dcc70f086817d86109947b231fb not found: ID does not exist"
Sep 06 19:54:07.627470 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.627403    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:54:07.663463 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.663435    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcfgn\" (UniqueName: \"kubernetes.io/projected/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-kube-api-access-jcfgn\") pod \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") "
Sep 06 19:54:07.663463 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.663466    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-home\") pod \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") "
Sep 06 19:54:07.663719 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.663496    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-model-cache\") pod \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") "
Sep 06 19:54:07.663719 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.663556    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-dshm\") pod \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") "
Sep 06 19:54:07.663719 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.663604    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-tls-certs\") pod \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") "
Sep 06 19:54:07.663719 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.663642    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-tmp-dir\") pod \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\" (UID: \"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c\") "
Sep 06 19:54:07.663945 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.663752    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-home" (OuterVolumeSpecName: "home") pod "630cf613-76cb-44a7-94e7-5bb1d4d4dd0c" (UID: "630cf613-76cb-44a7-94e7-5bb1d4d4dd0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:07.663945 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.663774    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "630cf613-76cb-44a7-94e7-5bb1d4d4dd0c" (UID: "630cf613-76cb-44a7-94e7-5bb1d4d4dd0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:07.664076 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.663990    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "630cf613-76cb-44a7-94e7-5bb1d4d4dd0c" (UID: "630cf613-76cb-44a7-94e7-5bb1d4d4dd0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:07.664076 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.664069    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-tmp-dir\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:07.664185 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.664084    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-home\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:07.664185 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.664094    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-model-cache\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:07.665656 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.665628    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-dshm" (OuterVolumeSpecName: "dshm") pod "630cf613-76cb-44a7-94e7-5bb1d4d4dd0c" (UID: "630cf613-76cb-44a7-94e7-5bb1d4d4dd0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:07.665872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.665852    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-kube-api-access-jcfgn" (OuterVolumeSpecName: "kube-api-access-jcfgn") pod "630cf613-76cb-44a7-94e7-5bb1d4d4dd0c" (UID: "630cf613-76cb-44a7-94e7-5bb1d4d4dd0c"). InnerVolumeSpecName "kube-api-access-jcfgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:54:07.665936 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.665905    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "630cf613-76cb-44a7-94e7-5bb1d4d4dd0c" (UID: "630cf613-76cb-44a7-94e7-5bb1d4d4dd0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:54:07.764783 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.764743    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-dshm\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:07.764783 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.764776    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-tls-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:07.764783 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:07.764787    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcfgn\" (UniqueName: \"kubernetes.io/projected/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c-kube-api-access-jcfgn\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:08.463313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:08.463276    2571 generic.go:356] "Generic (PLEG): container finished" podID="630cf613-76cb-44a7-94e7-5bb1d4d4dd0c" containerID="335c4742ee7af67eacd81003f66f226a040acbcc324702e790f2ef82dcb4dda1" exitCode=0
Sep 06 19:54:08.463810 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:08.463326    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"
Sep 06 19:54:08.463810 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:08.463355    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq" event={"ID":"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c","Type":"ContainerDied","Data":"335c4742ee7af67eacd81003f66f226a040acbcc324702e790f2ef82dcb4dda1"}
Sep 06 19:54:08.463810 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:08.463391    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq" event={"ID":"630cf613-76cb-44a7-94e7-5bb1d4d4dd0c","Type":"ContainerDied","Data":"3c93d39cf0365ce346d6a2352642cb267db50711704590a4338ab1cefb2259d2"}
Sep 06 19:54:08.463810 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:08.463409    2571 scope.go:117] "RemoveContainer" containerID="335c4742ee7af67eacd81003f66f226a040acbcc324702e790f2ef82dcb4dda1"
Sep 06 19:54:08.473020 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:08.472988    2571 scope.go:117] "RemoveContainer" containerID="335c4742ee7af67eacd81003f66f226a040acbcc324702e790f2ef82dcb4dda1"
Sep 06 19:54:08.473364 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:54:08.473335    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335c4742ee7af67eacd81003f66f226a040acbcc324702e790f2ef82dcb4dda1\": container with ID starting with 335c4742ee7af67eacd81003f66f226a040acbcc324702e790f2ef82dcb4dda1 not found: ID does not exist" containerID="335c4742ee7af67eacd81003f66f226a040acbcc324702e790f2ef82dcb4dda1"
Sep 06 19:54:08.473476 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:08.473374    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335c4742ee7af67eacd81003f66f226a040acbcc324702e790f2ef82dcb4dda1"} err="failed to get container status \"335c4742ee7af67eacd81003f66f226a040acbcc324702e790f2ef82dcb4dda1\": rpc error: code = NotFound desc = could not find container \"335c4742ee7af67eacd81003f66f226a040acbcc324702e790f2ef82dcb4dda1\": container with ID starting with 335c4742ee7af67eacd81003f66f226a040acbcc324702e790f2ef82dcb4dda1 not found: ID does not exist"
Sep 06 19:54:08.485616 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:08.485587    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"]
Sep 06 19:54:08.489078 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:08.489050    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-712616/rate-test-model-712616-kserve-65d4dff6c9-gz8wq"]
Sep 06 19:54:08.919411 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:08.919338    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="630cf613-76cb-44a7-94e7-5bb1d4d4dd0c" path="/var/lib/kubelet/pods/630cf613-76cb-44a7-94e7-5bb1d4d4dd0c/volumes"
Sep 06 19:54:08.919802 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:08.919779    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80e927b7-0065-4f5b-8299-a2e37dff3ccd" path="/var/lib/kubelet/pods/80e927b7-0065-4f5b-8299-a2e37dff3ccd/volumes"
Sep 06 19:54:17.231101 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:17.231068    2571 scope.go:117] "RemoveContainer" containerID="a25afd14c697bdad643f7d9b68e9a92cb3d0b36560061004eec0ee73cf5e8588"
Sep 06 19:54:17.240085 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:17.240059    2571 scope.go:117] "RemoveContainer" containerID="8e4dcb21bb5de7b00bb5b0bba98d4910de49c744196d3d266ad543453dd9294e"
Sep 06 19:54:19.135487 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:19.135448    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:54:22.172357 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172318    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"]
Sep 06 19:54:22.172843 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172776    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="618a6f65-2d10-4cc6-8fe3-ec941aff03d1" containerName="maas-api"
Sep 06 19:54:22.172843 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172789    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="618a6f65-2d10-4cc6-8fe3-ec941aff03d1" containerName="maas-api"
Sep 06 19:54:22.172843 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172808    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce637caa-427f-47a6-af77-6efff8fe023e" containerName="istio-proxy"
Sep 06 19:54:22.172843 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172815    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce637caa-427f-47a6-af77-6efff8fe023e" containerName="istio-proxy"
Sep 06 19:54:22.172843 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172836    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce637caa-427f-47a6-af77-6efff8fe023e" containerName="istio-proxy"
Sep 06 19:54:22.172843 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172842    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce637caa-427f-47a6-af77-6efff8fe023e" containerName="istio-proxy"
Sep 06 19:54:22.173205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172851    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="630cf613-76cb-44a7-94e7-5bb1d4d4dd0c" containerName="main"
Sep 06 19:54:22.173205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172857    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="630cf613-76cb-44a7-94e7-5bb1d4d4dd0c" containerName="main"
Sep 06 19:54:22.173205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172866    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" containerName="istio-proxy"
Sep 06 19:54:22.173205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172871    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" containerName="istio-proxy"
Sep 06 19:54:22.173205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172878    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e927b7-0065-4f5b-8299-a2e37dff3ccd" containerName="main"
Sep 06 19:54:22.173205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172883    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e927b7-0065-4f5b-8299-a2e37dff3ccd" containerName="main"
Sep 06 19:54:22.173205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172891    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" containerName="istio-proxy"
Sep 06 19:54:22.173205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172899    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" containerName="istio-proxy"
Sep 06 19:54:22.173205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172969    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e927b7-0065-4f5b-8299-a2e37dff3ccd" containerName="main"
Sep 06 19:54:22.173205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172980    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" containerName="istio-proxy"
Sep 06 19:54:22.173205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172987    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce637caa-427f-47a6-af77-6efff8fe023e" containerName="istio-proxy"
Sep 06 19:54:22.173205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.172993    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="630cf613-76cb-44a7-94e7-5bb1d4d4dd0c" containerName="main"
Sep 06 19:54:22.173205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.173003    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce637caa-427f-47a6-af77-6efff8fe023e" containerName="istio-proxy"
Sep 06 19:54:22.173205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.173009    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="618a6f65-2d10-4cc6-8fe3-ec941aff03d1" containerName="maas-api"
Sep 06 19:54:22.176291 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.176268    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.178778 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.178751    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-adaf11-openshift-default-dockercfg-8bn7k\""
Sep 06 19:54:22.188244 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.188215    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"]
Sep 06 19:54:22.306453 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.306405    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-workload-socket\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.306637 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.306481    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-podinfo\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.306637 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.306524    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-envoy\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.306637 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.306559    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-workload-certs\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.306637 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.306589    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-token\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.306779 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.306689    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfbb2\" (UniqueName: \"kubernetes.io/projected/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-kube-api-access-kfbb2\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.306779 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.306725    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istiod-ca-cert\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.306850 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.306791    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-credential-socket\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.306850 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.306822    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-data\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.408370 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.408331    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-podinfo\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.408557 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.408383    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-envoy\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.408557 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.408484    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-workload-certs\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.408557 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.408506    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-token\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.408722 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.408558    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfbb2\" (UniqueName: \"kubernetes.io/projected/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-kube-api-access-kfbb2\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.408722 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.408679    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istiod-ca-cert\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.408854 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.408743    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-credential-socket\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.408854 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.408772    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-data\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.408854 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.408816    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-workload-socket\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.409355 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.409328    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-workload-socket\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.409355 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.409349    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-workload-certs\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.409539 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.409466    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-data\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.409643 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.409616    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istiod-ca-cert\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.409746 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.409702    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-credential-socket\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.411594 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.411562    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-podinfo\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.411769 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.411753    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-envoy\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.422676 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.422615    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfbb2\" (UniqueName: \"kubernetes.io/projected/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-kube-api-access-kfbb2\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.422807 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.422739    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-token\") pod \"e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") " pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.487946 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.487903    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:22.766227 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.766202    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"]
Sep 06 19:54:22.768826 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:54:22.768795    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ad7d2f_e606_4275_981f_31f9ff66c8e7.slice/crio-d4948508ac519522a6077ce0ab6d3742a41951123f869dff0c18aab5f553dcd7 WatchSource:0}: Error finding container d4948508ac519522a6077ce0ab6d3742a41951123f869dff0c18aab5f553dcd7: Status 404 returned error can't find the container with id d4948508ac519522a6077ce0ab6d3742a41951123f869dff0c18aab5f553dcd7
Sep 06 19:54:22.771219 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.771184    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:54:22.771294 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.771256    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:54:22.771294 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:22.771285    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:54:23.546995 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:23.546946    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq" event={"ID":"e8ad7d2f-e606-4275-981f-31f9ff66c8e7","Type":"ContainerStarted","Data":"79223ab9010b3ed3bdfa61d68e164ccb3f43aab6500fe604c40bc86468b2ee6c"}
Sep 06 19:54:23.547429 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:23.547021    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq" event={"ID":"e8ad7d2f-e606-4275-981f-31f9ff66c8e7","Type":"ContainerStarted","Data":"d4948508ac519522a6077ce0ab6d3742a41951123f869dff0c18aab5f553dcd7"}
Sep 06 19:54:23.565141 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:23.565092    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq" podStartSLOduration=1.565073993 podStartE2EDuration="1.565073993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:54:23.563416269 +0000 UTC m=+1627.133714615" watchObservedRunningTime="2026-09-06 19:54:23.565073993 +0000 UTC m=+1627.135372339"
Sep 06 19:54:24.227880 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.227837    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"]
Sep 06 19:54:24.228153 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.228121    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp" podUID="f498a7d9-d881-4050-8d12-86684816e5fb" containerName="maas-api" containerID="cri-o://b8d1c78473cf1ea420b97ef75e954fa4da52754861a5f2113c2c520f3effc27c" gracePeriod=30
Sep 06 19:54:24.479484 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.479417    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:54:24.489097 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.489072    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:24.493871 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.493845    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:24.554435 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.554398    2571 generic.go:356] "Generic (PLEG): container finished" podID="f498a7d9-d881-4050-8d12-86684816e5fb" containerID="b8d1c78473cf1ea420b97ef75e954fa4da52754861a5f2113c2c520f3effc27c" exitCode=0
Sep 06 19:54:24.554877 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.554459    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"
Sep 06 19:54:24.554877 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.554484    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp" event={"ID":"f498a7d9-d881-4050-8d12-86684816e5fb","Type":"ContainerDied","Data":"b8d1c78473cf1ea420b97ef75e954fa4da52754861a5f2113c2c520f3effc27c"}
Sep 06 19:54:24.554877 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.554542    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp" event={"ID":"f498a7d9-d881-4050-8d12-86684816e5fb","Type":"ContainerDied","Data":"8dbe1d71dfee4d34365350b50f14e5d84f810febe434537e516e9f734e20eca5"}
Sep 06 19:54:24.554877 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.554566    2571 scope.go:117] "RemoveContainer" containerID="b8d1c78473cf1ea420b97ef75e954fa4da52754861a5f2113c2c520f3effc27c"
Sep 06 19:54:24.555099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.555002    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:24.556454 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.556300    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:24.564446 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.564410    2571 scope.go:117] "RemoveContainer" containerID="b8d1c78473cf1ea420b97ef75e954fa4da52754861a5f2113c2c520f3effc27c"
Sep 06 19:54:24.564799 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:54:24.564774    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d1c78473cf1ea420b97ef75e954fa4da52754861a5f2113c2c520f3effc27c\": container with ID starting with b8d1c78473cf1ea420b97ef75e954fa4da52754861a5f2113c2c520f3effc27c not found: ID does not exist" containerID="b8d1c78473cf1ea420b97ef75e954fa4da52754861a5f2113c2c520f3effc27c"
Sep 06 19:54:24.564877 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.564815    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d1c78473cf1ea420b97ef75e954fa4da52754861a5f2113c2c520f3effc27c"} err="failed to get container status \"b8d1c78473cf1ea420b97ef75e954fa4da52754861a5f2113c2c520f3effc27c\": rpc error: code = NotFound desc = could not find container \"b8d1c78473cf1ea420b97ef75e954fa4da52754861a5f2113c2c520f3effc27c\": container with ID starting with b8d1c78473cf1ea420b97ef75e954fa4da52754861a5f2113c2c520f3effc27c not found: ID does not exist"
Sep 06 19:54:24.635002 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.634953    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f498a7d9-d881-4050-8d12-86684816e5fb-maas-api-tls\") pod \"f498a7d9-d881-4050-8d12-86684816e5fb\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") "
Sep 06 19:54:24.635157 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.635060    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f498a7d9-d881-4050-8d12-86684816e5fb-openshift-service-ca\") pod \"f498a7d9-d881-4050-8d12-86684816e5fb\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") "
Sep 06 19:54:24.635157 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.635088    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvzcq\" (UniqueName: \"kubernetes.io/projected/f498a7d9-d881-4050-8d12-86684816e5fb-kube-api-access-dvzcq\") pod \"f498a7d9-d881-4050-8d12-86684816e5fb\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") "
Sep 06 19:54:24.635157 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.635130    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f498a7d9-d881-4050-8d12-86684816e5fb-metrics-certs\") pod \"f498a7d9-d881-4050-8d12-86684816e5fb\" (UID: \"f498a7d9-d881-4050-8d12-86684816e5fb\") "
Sep 06 19:54:24.635499 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.635464    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f498a7d9-d881-4050-8d12-86684816e5fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f498a7d9-d881-4050-8d12-86684816e5fb" (UID: "f498a7d9-d881-4050-8d12-86684816e5fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:54:24.637282 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.637253    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f498a7d9-d881-4050-8d12-86684816e5fb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f498a7d9-d881-4050-8d12-86684816e5fb" (UID: "f498a7d9-d881-4050-8d12-86684816e5fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:54:24.637381 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.637332    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f498a7d9-d881-4050-8d12-86684816e5fb-kube-api-access-dvzcq" (OuterVolumeSpecName: "kube-api-access-dvzcq") pod "f498a7d9-d881-4050-8d12-86684816e5fb" (UID: "f498a7d9-d881-4050-8d12-86684816e5fb"). InnerVolumeSpecName "kube-api-access-dvzcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:54:24.637433 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.637381    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f498a7d9-d881-4050-8d12-86684816e5fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f498a7d9-d881-4050-8d12-86684816e5fb" (UID: "f498a7d9-d881-4050-8d12-86684816e5fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:54:24.737055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.736940    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f498a7d9-d881-4050-8d12-86684816e5fb-maas-api-tls\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:24.737055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.736989    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f498a7d9-d881-4050-8d12-86684816e5fb-openshift-service-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:24.737055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.737002    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvzcq\" (UniqueName: \"kubernetes.io/projected/f498a7d9-d881-4050-8d12-86684816e5fb-kube-api-access-dvzcq\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:24.737055 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.737011    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f498a7d9-d881-4050-8d12-86684816e5fb-metrics-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:24.877417 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.877376    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"]
Sep 06 19:54:24.879633 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.879608    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6ddeb5-756898fbff-bzhhp"]
Sep 06 19:54:24.918667 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:24.918631    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f498a7d9-d881-4050-8d12-86684816e5fb" path="/var/lib/kubelet/pods/f498a7d9-d881-4050-8d12-86684816e5fb/volumes"
Sep 06 19:54:26.745881 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.745812    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"]
Sep 06 19:54:26.746489 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.746466    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f498a7d9-d881-4050-8d12-86684816e5fb" containerName="maas-api"
Sep 06 19:54:26.746654 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.746640    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f498a7d9-d881-4050-8d12-86684816e5fb" containerName="maas-api"
Sep 06 19:54:26.746843 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.746824    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="558eb546-2a5e-4d73-ac28-0c3a07a5bd4e" containerName="istio-proxy"
Sep 06 19:54:26.746928 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.746848    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f498a7d9-d881-4050-8d12-86684816e5fb" containerName="maas-api"
Sep 06 19:54:26.751517 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.751496    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:26.766423 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.766385    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 06 19:54:26.766735 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.766718    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-adaf11-serving-cert\""
Sep 06 19:54:26.767002 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.766981    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bfdfp\""
Sep 06 19:54:26.767296 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.767280    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 19:54:26.770148 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.767508    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 06 19:54:26.772680 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.772609    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"]
Sep 06 19:54:26.863985 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.863918    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-metrics-certs\") pod \"maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:26.864208 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.864009    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfc28\" (UniqueName: \"kubernetes.io/projected/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-kube-api-access-pfc28\") pod \"maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:26.864208 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.864105    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:26.864208 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.864162    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:26.965666 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.965616    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-metrics-certs\") pod \"maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:26.965666 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.965669    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfc28\" (UniqueName: \"kubernetes.io/projected/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-kube-api-access-pfc28\") pod \"maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:26.965925 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.965746    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:26.965925 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.965798    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:26.966475 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.966450    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:26.968072 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.968048    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:26.968205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.968185    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-metrics-certs\") pod \"maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:26.973157 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:26.973134    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfc28\" (UniqueName: \"kubernetes.io/projected/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-kube-api-access-pfc28\") pod \"maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:27.079849 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:27.079807    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:27.356029 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:27.356003    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"]
Sep 06 19:54:27.358174 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:54:27.358136    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b2b07e_3f36_4c55_9b4d_a115c816aaa3.slice/crio-ab35f95ff3a98ece7bcee694a37f089161926db448b7da76cba11ec54ff03acd WatchSource:0}: Error finding container ab35f95ff3a98ece7bcee694a37f089161926db448b7da76cba11ec54ff03acd: Status 404 returned error can't find the container with id ab35f95ff3a98ece7bcee694a37f089161926db448b7da76cba11ec54ff03acd
Sep 06 19:54:27.571880 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:27.571845    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn" event={"ID":"24b2b07e-3f36-4c55-9b4d-a115c816aaa3","Type":"ContainerStarted","Data":"ab35f95ff3a98ece7bcee694a37f089161926db448b7da76cba11ec54ff03acd"}
Sep 06 19:54:28.578685 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:28.578646    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn" event={"ID":"24b2b07e-3f36-4c55-9b4d-a115c816aaa3","Type":"ContainerStarted","Data":"0d5dd920c7905de1a5bc82bd39f99e59758039f9be10b57dd0d8e16f0f1983d9"}
Sep 06 19:54:28.579093 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:28.578729    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:28.597023 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:28.596969    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn" podStartSLOduration=2.24872755 podStartE2EDuration="2.596938824s" totalImagesPullingTime="348.211274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:54:28.594275661 +0000 UTC m=+1632.164574008" watchObservedRunningTime="2026-09-06 19:54:28.596938824 +0000 UTC m=+1632.167237171"
Sep 06 19:54:30.573329 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:30.573291    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"]
Sep 06 19:54:30.573763 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:30.573548    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq" podUID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" containerName="istio-proxy" containerID="cri-o://2b62ef1f6e7e637e9c69319d88a090e47253a4824d7be14ef4e99652cfcade1f" gracePeriod=30
Sep 06 19:54:33.939644 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:33.939602    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:54:34.590762 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:34.590731    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:54:35.867745 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.867676    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:54:35.960473 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.960440    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-token\") pod \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") "
Sep 06 19:54:35.960662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.960492    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-podinfo\") pod \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") "
Sep 06 19:54:35.960662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.960527    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-workload-socket\") pod \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") "
Sep 06 19:54:35.960662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.960573    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw5lh\" (UniqueName: \"kubernetes.io/projected/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-kube-api-access-fw5lh\") pod \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") "
Sep 06 19:54:35.960662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.960602    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-credential-socket\") pod \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") "
Sep 06 19:54:35.960662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.960654    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-data\") pod \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") "
Sep 06 19:54:35.960950 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.960685    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istiod-ca-cert\") pod \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") "
Sep 06 19:54:35.960950 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.960727    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-workload-certs\") pod \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") "
Sep 06 19:54:35.960950 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.960762    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-envoy\") pod \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\" (UID: \"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961\") "
Sep 06 19:54:35.960950 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.960825    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" (UID: "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:35.960950 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.960914    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" (UID: "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:35.960950 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.960928    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-data" (OuterVolumeSpecName: "istio-data") pod "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" (UID: "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:35.961200 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.961068    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" (UID: "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:35.961200 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.961128    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" (UID: "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:54:35.961285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.961235    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-workload-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:35.961285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.961247    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-credential-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:35.961285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.961258    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-data\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:35.961285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.961272    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istiod-ca-cert\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:35.961285 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.961284    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-workload-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:35.962993 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.962942    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-token" (OuterVolumeSpecName: "istio-token") pod "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" (UID: "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:54:35.963081 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.963006    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-kube-api-access-fw5lh" (OuterVolumeSpecName: "kube-api-access-fw5lh") pod "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" (UID: "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961"). InnerVolumeSpecName "kube-api-access-fw5lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:54:35.963192 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.963171    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" (UID: "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:54:35.963276 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:35.963260    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" (UID: "1b62b4d1-ecd0-47b2-b79b-01d1b4bef961"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 19:54:36.062150 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.062112    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-token\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:36.062150 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.062143    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-podinfo\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:36.062150 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.062153    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw5lh\" (UniqueName: \"kubernetes.io/projected/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-kube-api-access-fw5lh\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:36.062150 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.062162    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961-istio-envoy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:54:36.623224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.623193    2571 generic.go:356] "Generic (PLEG): container finished" podID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" containerID="2b62ef1f6e7e637e9c69319d88a090e47253a4824d7be14ef4e99652cfcade1f" exitCode=0
Sep 06 19:54:36.623410 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.623242    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"
Sep 06 19:54:36.623410 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.623253    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq" event={"ID":"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961","Type":"ContainerDied","Data":"2b62ef1f6e7e637e9c69319d88a090e47253a4824d7be14ef4e99652cfcade1f"}
Sep 06 19:54:36.623410 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.623299    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq" event={"ID":"1b62b4d1-ecd0-47b2-b79b-01d1b4bef961","Type":"ContainerDied","Data":"a528b87250a3082899906bbef34079ff56fd0a55ecf1b2338e2b28ec53b17824"}
Sep 06 19:54:36.623410 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.623321    2571 scope.go:117] "RemoveContainer" containerID="2b62ef1f6e7e637e9c69319d88a090e47253a4824d7be14ef4e99652cfcade1f"
Sep 06 19:54:36.633217 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.633189    2571 scope.go:117] "RemoveContainer" containerID="99c9f2af719ef054a1b3aa6e1cb06aa92cc97c2f48dfcfebb6a3ff516b024977"
Sep 06 19:54:36.650016 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.649982    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"]
Sep 06 19:54:36.654398 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.654372    2571 scope.go:117] "RemoveContainer" containerID="2b62ef1f6e7e637e9c69319d88a090e47253a4824d7be14ef4e99652cfcade1f"
Sep 06 19:54:36.655086 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:54:36.655057    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b62ef1f6e7e637e9c69319d88a090e47253a4824d7be14ef4e99652cfcade1f\": container with ID starting with 2b62ef1f6e7e637e9c69319d88a090e47253a4824d7be14ef4e99652cfcade1f not found: ID does not exist" containerID="2b62ef1f6e7e637e9c69319d88a090e47253a4824d7be14ef4e99652cfcade1f"
Sep 06 19:54:36.655205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.655096    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b62ef1f6e7e637e9c69319d88a090e47253a4824d7be14ef4e99652cfcade1f"} err="failed to get container status \"2b62ef1f6e7e637e9c69319d88a090e47253a4824d7be14ef4e99652cfcade1f\": rpc error: code = NotFound desc = could not find container \"2b62ef1f6e7e637e9c69319d88a090e47253a4824d7be14ef4e99652cfcade1f\": container with ID starting with 2b62ef1f6e7e637e9c69319d88a090e47253a4824d7be14ef4e99652cfcade1f not found: ID does not exist"
Sep 06 19:54:36.655205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.655122    2571 scope.go:117] "RemoveContainer" containerID="99c9f2af719ef054a1b3aa6e1cb06aa92cc97c2f48dfcfebb6a3ff516b024977"
Sep 06 19:54:36.655205 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.655147    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ddeb5-openshift-default-95755b49b-g5xbq"]
Sep 06 19:54:36.655437 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:54:36.655410    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c9f2af719ef054a1b3aa6e1cb06aa92cc97c2f48dfcfebb6a3ff516b024977\": container with ID starting with 99c9f2af719ef054a1b3aa6e1cb06aa92cc97c2f48dfcfebb6a3ff516b024977 not found: ID does not exist" containerID="99c9f2af719ef054a1b3aa6e1cb06aa92cc97c2f48dfcfebb6a3ff516b024977"
Sep 06 19:54:36.655502 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.655445    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c9f2af719ef054a1b3aa6e1cb06aa92cc97c2f48dfcfebb6a3ff516b024977"} err="failed to get container status \"99c9f2af719ef054a1b3aa6e1cb06aa92cc97c2f48dfcfebb6a3ff516b024977\": rpc error: code = NotFound desc = could not find container \"99c9f2af719ef054a1b3aa6e1cb06aa92cc97c2f48dfcfebb6a3ff516b024977\": container with ID starting with 99c9f2af719ef054a1b3aa6e1cb06aa92cc97c2f48dfcfebb6a3ff516b024977 not found: ID does not exist"
Sep 06 19:54:36.919150 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:36.919066    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" path="/var/lib/kubelet/pods/1b62b4d1-ecd0-47b2-b79b-01d1b4bef961/volumes"
Sep 06 19:54:43.668223 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:43.668190    2571 generic.go:356] "Generic (PLEG): container finished" podID="e8ad7d2f-e606-4275-981f-31f9ff66c8e7" containerID="79223ab9010b3ed3bdfa61d68e164ccb3f43aab6500fe604c40bc86468b2ee6c" exitCode=0
Sep 06 19:54:43.668638 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:43.668232    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq" event={"ID":"e8ad7d2f-e606-4275-981f-31f9ff66c8e7","Type":"ContainerDied","Data":"79223ab9010b3ed3bdfa61d68e164ccb3f43aab6500fe604c40bc86468b2ee6c"}
Sep 06 19:54:43.668638 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:43.668594    2571 scope.go:117] "RemoveContainer" containerID="79223ab9010b3ed3bdfa61d68e164ccb3f43aab6500fe604c40bc86468b2ee6c"
Sep 06 19:54:43.672367 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:43.670278    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:54:43.672367 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:43.670450    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:54:43.672367 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:43.670609    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:54:44.675218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:44.675179    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq" event={"ID":"e8ad7d2f-e606-4275-981f-31f9ff66c8e7","Type":"ContainerStarted","Data":"4c7396cc3005c45723bc8cafa87d29f3cf2fa1ee9260074e3a0af1a16b9b18f4"}
Sep 06 19:54:45.488824 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:45.488778    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:45.489105 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:45.489080    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 06 19:54:45.489192 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:45.489139    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq" podUID="e8ad7d2f-e606-4275-981f-31f9ff66c8e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 06 19:54:46.489228 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:46.489191    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 06 19:54:46.489624 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:46.489262    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq" podUID="e8ad7d2f-e606-4275-981f-31f9ff66c8e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 06 19:54:47.488718 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:47.488681    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 06 19:54:47.488921 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:47.488750    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq" podUID="e8ad7d2f-e606-4275-981f-31f9ff66c8e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 06 19:54:48.488407 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:48.488367    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 06 19:54:48.488799 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:48.488427    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq" podUID="e8ad7d2f-e606-4275-981f-31f9ff66c8e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 06 19:54:49.488576 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:49.488537    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 06 19:54:49.489099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:49.488601    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq" podUID="e8ad7d2f-e606-4275-981f-31f9ff66c8e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 06 19:54:50.492904 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:50.492872    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:50.493306 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:50.493136    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:54:50.493904 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:54:50.493885    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:55:06.145342 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.145306    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"]
Sep 06 19:55:06.145744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.145730    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" containerName="istio-proxy"
Sep 06 19:55:06.145790 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.145746    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" containerName="istio-proxy"
Sep 06 19:55:06.145828 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.145819    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" containerName="istio-proxy"
Sep 06 19:55:06.145862 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.145829    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" containerName="istio-proxy"
Sep 06 19:55:06.149175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.149157    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.152744 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.152719    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a30d6c\"/\"kube-root-ca.crt\""
Sep 06 19:55:06.152881 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.152759    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a30d6c\"/\"openshift-service-ca.crt\""
Sep 06 19:55:06.152881 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.152777    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a30d6c\"/\"default-dockercfg-v74h8\""
Sep 06 19:55:06.152881 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.152758    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a30d6c\"/\"ipp-route-a30d6c-kserve-self-signed-certs\""
Sep 06 19:55:06.157662 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.157640    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"]
Sep 06 19:55:06.265423 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.265382    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-home\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.265604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.265429    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3883d3ad-5748-4d31-8b92-7ef374ba5da4-tls-certs\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.265604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.265470    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-model-cache\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.265604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.265542    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-dshm\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.265604 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.265580    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-tmp-dir\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.265764 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.265616    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94vm6\" (UniqueName: \"kubernetes.io/projected/3883d3ad-5748-4d31-8b92-7ef374ba5da4-kube-api-access-94vm6\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.366428 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.366387    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-model-cache\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.366616 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.366450    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-dshm\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.366664 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.366619    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-tmp-dir\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.366707 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.366684    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94vm6\" (UniqueName: \"kubernetes.io/projected/3883d3ad-5748-4d31-8b92-7ef374ba5da4-kube-api-access-94vm6\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.366818 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.366801    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-home\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.366895 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.366835    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3883d3ad-5748-4d31-8b92-7ef374ba5da4-tls-certs\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.366895 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.366864    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-model-cache\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.367022 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.366952    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-tmp-dir\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.367138 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.367117    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-home\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.368777 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.368752    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-dshm\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.369210 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.369195    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3883d3ad-5748-4d31-8b92-7ef374ba5da4-tls-certs\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.374127 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.374103    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94vm6\" (UniqueName: \"kubernetes.io/projected/3883d3ad-5748-4d31-8b92-7ef374ba5da4-kube-api-access-94vm6\") pod \"ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") " pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.459657 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.459571    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:06.729199 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.729171    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"]
Sep 06 19:55:06.730556 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:55:06.730526    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3883d3ad_5748_4d31_8b92_7ef374ba5da4.slice/crio-9e101c678492a62effd91c6b05b0f26c958169876c692ba4d283f9227184daff WatchSource:0}: Error finding container 9e101c678492a62effd91c6b05b0f26c958169876c692ba4d283f9227184daff: Status 404 returned error can't find the container with id 9e101c678492a62effd91c6b05b0f26c958169876c692ba4d283f9227184daff
Sep 06 19:55:06.791044 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:06.791013    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2" event={"ID":"3883d3ad-5748-4d31-8b92-7ef374ba5da4","Type":"ContainerStarted","Data":"9e101c678492a62effd91c6b05b0f26c958169876c692ba4d283f9227184daff"}
Sep 06 19:55:07.798442 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:07.798407    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2" event={"ID":"3883d3ad-5748-4d31-8b92-7ef374ba5da4","Type":"ContainerStarted","Data":"f110154ff05bb92077722f1419aed757cb7bef004cbaf7bba86fed0731e5152c"}
Sep 06 19:55:07.815137 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:07.815084    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2" podStartSLOduration=1.8150683399999998 podStartE2EDuration="1.81506834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:55:07.812820699 +0000 UTC m=+1671.383119045" watchObservedRunningTime="2026-09-06 19:55:07.81506834 +0000 UTC m=+1671.385366686"
Sep 06 19:55:09.049569 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:09.049528    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:55:16.460294 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:16.460255    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:16.460768 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:16.460309    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:16.473072 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:16.473039    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:16.854680 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:16.854653    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:55:45.745106 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:45.745015    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:55:52.142935 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.142889    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"]
Sep 06 19:55:52.143488 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.143471    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" containerName="istio-proxy"
Sep 06 19:55:52.143556 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.143489    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b62b4d1-ecd0-47b2-b79b-01d1b4bef961" containerName="istio-proxy"
Sep 06 19:55:52.147267 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.147239    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.149822 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.149796    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-c95664-openshift-default-dockercfg-jwg4k\""
Sep 06 19:55:52.158298 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.157617    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"]
Sep 06 19:55:52.215547 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.215487    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-token\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.215748 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.215575    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-workload-socket\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.215748 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.215606    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rcdp\" (UniqueName: \"kubernetes.io/projected/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-kube-api-access-9rcdp\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.215748 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.215647    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-podinfo\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.215748 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.215669    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-credential-socket\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.215748 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.215692    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.215748 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.215747    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-workload-certs\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.216065 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.215791    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-data\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.216065 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.215855    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-envoy\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.316970 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.316904    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-envoy\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.317177 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.317004    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-token\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.317177 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.317065    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-workload-socket\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.317177 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.317092    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rcdp\" (UniqueName: \"kubernetes.io/projected/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-kube-api-access-9rcdp\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.317177 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.317136    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-podinfo\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.317177 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.317159    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-credential-socket\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.317449 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.317182    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.317449 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.317236    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-workload-certs\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.317449 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.317271    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-data\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.317652 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.317627    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-data\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.318028 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.318005    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-workload-socket\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.318793 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.318061    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-credential-socket\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.318921 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.318640    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.319076 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.318747    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-workload-certs\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.320438 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.320411    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-podinfo\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.321549 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.321525    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-envoy\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.326445 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.326420    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-token\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.326899 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.326855    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rcdp\" (UniqueName: \"kubernetes.io/projected/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-kube-api-access-9rcdp\") pod \"e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") " pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.461924 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.461823    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:52.738550 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.738501    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"]
Sep 06 19:55:52.741662 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:55:52.741618    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3c7e71_38c7_48d2_8946_6a27aa5e14bc.slice/crio-cebc1014f96311bfc7e8a98875f4b686e64073d89692a8fce23773afcea71d48 WatchSource:0}: Error finding container cebc1014f96311bfc7e8a98875f4b686e64073d89692a8fce23773afcea71d48: Status 404 returned error can't find the container with id cebc1014f96311bfc7e8a98875f4b686e64073d89692a8fce23773afcea71d48
Sep 06 19:55:52.743930 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.743897    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:55:52.744056 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.743984    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:55:52.744056 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:52.744019    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:55:53.030466 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:53.030381    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt" event={"ID":"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc","Type":"ContainerStarted","Data":"6fa5739f5a0bfa55c4ed1f4a47d286a613bcde5af0f43281ce5d2ea0cae80260"}
Sep 06 19:55:53.030466 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:53.030439    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt" event={"ID":"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc","Type":"ContainerStarted","Data":"cebc1014f96311bfc7e8a98875f4b686e64073d89692a8fce23773afcea71d48"}
Sep 06 19:55:53.049553 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:53.049500    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt" podStartSLOduration=1.049483145 podStartE2EDuration="1.049483145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:55:53.046840676 +0000 UTC m=+1716.617139034" watchObservedRunningTime="2026-09-06 19:55:53.049483145 +0000 UTC m=+1716.619781491"
Sep 06 19:55:53.462859 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:53.462815    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:53.468563 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:53.468534    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:54.036210 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:54.036177    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:54.037226 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:54.037203    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:55:56.037670 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.037635    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"]
Sep 06 19:55:56.041703 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.041674    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:55:56.044310 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.044284    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-c95664-serving-cert\""
Sep 06 19:55:56.053115 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.053086    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"]
Sep 06 19:55:56.158596 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.158564    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:55:56.158854 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.158654    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:55:56.158854 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.158796    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm2gh\" (UniqueName: \"kubernetes.io/projected/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-kube-api-access-bm2gh\") pod \"maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:55:56.159017 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.158850    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:55:56.259892 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.259849    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2gh\" (UniqueName: \"kubernetes.io/projected/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-kube-api-access-bm2gh\") pod \"maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:55:56.260119 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.259898    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:55:56.260119 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.259998    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:55:56.260119 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.260075    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:55:56.260661 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.260635    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:55:56.262559 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.262538    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:55:56.262711 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.262690    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:55:56.267508 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.267483    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm2gh\" (UniqueName: \"kubernetes.io/projected/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-kube-api-access-bm2gh\") pod \"maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:55:56.356659 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.356616    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:55:56.631487 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:56.631459    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"]
Sep 06 19:55:56.633009 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:55:56.632977    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad2b114_6a8e_4b73_8721_b9b11ce1b23e.slice/crio-0c1e71c8aa60eaa60262cdce21a4c550650f45a85125b9a1e5f3805e7f19319c WatchSource:0}: Error finding container 0c1e71c8aa60eaa60262cdce21a4c550650f45a85125b9a1e5f3805e7f19319c: Status 404 returned error can't find the container with id 0c1e71c8aa60eaa60262cdce21a4c550650f45a85125b9a1e5f3805e7f19319c
Sep 06 19:55:57.053781 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:57.053703    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9" event={"ID":"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e","Type":"ContainerStarted","Data":"0c1e71c8aa60eaa60262cdce21a4c550650f45a85125b9a1e5f3805e7f19319c"}
Sep 06 19:55:58.060424 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:58.060389    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9" event={"ID":"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e","Type":"ContainerStarted","Data":"0366d7052cd6a88155d86cdc1d2dc7ae5049ac0f600d93e16814683c6b7f4d02"}
Sep 06 19:55:58.060823 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:58.060532    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:55:58.079149 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:55:58.079099    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9" podStartSLOduration=1.634734895 podStartE2EDuration="2.079082652s" totalImagesPullingTime="444.347757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:55:58.077617614 +0000 UTC m=+1721.647915961" watchObservedRunningTime="2026-09-06 19:55:58.079082652 +0000 UTC m=+1721.649381000"
Sep 06 19:56:03.438695 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:03.438658    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:56:04.071381 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:04.071345    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:56:14.144483 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:14.144447    2571 generic.go:356] "Generic (PLEG): container finished" podID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerID="6fa5739f5a0bfa55c4ed1f4a47d286a613bcde5af0f43281ce5d2ea0cae80260" exitCode=0
Sep 06 19:56:14.144920 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:14.144521    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt" event={"ID":"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc","Type":"ContainerDied","Data":"6fa5739f5a0bfa55c4ed1f4a47d286a613bcde5af0f43281ce5d2ea0cae80260"}
Sep 06 19:56:14.144991 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:14.144928    2571 scope.go:117] "RemoveContainer" containerID="6fa5739f5a0bfa55c4ed1f4a47d286a613bcde5af0f43281ce5d2ea0cae80260"
Sep 06 19:56:14.146126 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:14.146100    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:56:14.146198 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:14.146168    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:56:14.146235 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:14.146199    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 06 19:56:15.151573 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:15.151540    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt" event={"ID":"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc","Type":"ContainerStarted","Data":"7fd19e65a5dbcc9a8ab1d2f89aafd72ff9250b7b61ffce2e88e6add396f89382"}
Sep 06 19:56:15.463046 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:15.462924    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:56:15.463233 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:15.463203    2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 06 19:56:15.463334 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:15.463279    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt" podUID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 06 19:56:16.462804 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:16.462762    2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 06 19:56:16.463311 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:16.462821    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt" podUID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 06 19:56:17.462268 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:17.462222    2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 06 19:56:17.462455 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:17.462292    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt" podUID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 06 19:56:18.463029 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:18.462986    2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 06 19:56:18.463429 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:18.463051    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt" podUID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 06 19:56:19.462224 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:19.462185    2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 06 19:56:19.462414 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:19.462240    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt" podUID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 06 19:56:20.463045 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:20.462999    2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 06 19:56:20.463521 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:20.463070    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt" podUID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 06 19:56:21.467588 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:21.467554    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:56:21.468086 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:21.467983    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:56:21.468703 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:21.468685    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:56:26.520820 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:26.520780    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"]
Sep 06 19:56:26.521356 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:26.521008    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9" podUID="3ad2b114-6a8e-4b73-8721-b9b11ce1b23e" containerName="maas-api" containerID="cri-o://0366d7052cd6a88155d86cdc1d2dc7ae5049ac0f600d93e16814683c6b7f4d02" gracePeriod=30
Sep 06 19:56:26.763708 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:26.763677    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:56:26.867280 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:26.867245    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm2gh\" (UniqueName: \"kubernetes.io/projected/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-kube-api-access-bm2gh\") pod \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") "
Sep 06 19:56:26.867542 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:26.867300    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-openshift-service-ca\") pod \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") "
Sep 06 19:56:26.867542 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:26.867410    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-maas-api-tls\") pod \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") "
Sep 06 19:56:26.867542 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:26.867465    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-metrics-certs\") pod \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\" (UID: \"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e\") "
Sep 06 19:56:26.867726 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:26.867655    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ad2b114-6a8e-4b73-8721-b9b11ce1b23e" (UID: "3ad2b114-6a8e-4b73-8721-b9b11ce1b23e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:56:26.867832 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:26.867815    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-openshift-service-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:56:26.869516 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:26.869489    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3ad2b114-6a8e-4b73-8721-b9b11ce1b23e" (UID: "3ad2b114-6a8e-4b73-8721-b9b11ce1b23e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:56:26.869632 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:26.869545    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ad2b114-6a8e-4b73-8721-b9b11ce1b23e" (UID: "3ad2b114-6a8e-4b73-8721-b9b11ce1b23e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:56:26.869632 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:26.869554    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-kube-api-access-bm2gh" (OuterVolumeSpecName: "kube-api-access-bm2gh") pod "3ad2b114-6a8e-4b73-8721-b9b11ce1b23e" (UID: "3ad2b114-6a8e-4b73-8721-b9b11ce1b23e"). InnerVolumeSpecName "kube-api-access-bm2gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:56:26.969139 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:26.969100    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-maas-api-tls\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:56:26.969139 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:26.969131    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-metrics-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:56:26.969139 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:26.969140    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm2gh\" (UniqueName: \"kubernetes.io/projected/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e-kube-api-access-bm2gh\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:56:27.222284 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:27.222197    2571 generic.go:356] "Generic (PLEG): container finished" podID="3ad2b114-6a8e-4b73-8721-b9b11ce1b23e" containerID="0366d7052cd6a88155d86cdc1d2dc7ae5049ac0f600d93e16814683c6b7f4d02" exitCode=0
Sep 06 19:56:27.222284 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:27.222242    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"
Sep 06 19:56:27.222284 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:27.222272    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9" event={"ID":"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e","Type":"ContainerDied","Data":"0366d7052cd6a88155d86cdc1d2dc7ae5049ac0f600d93e16814683c6b7f4d02"}
Sep 06 19:56:27.222557 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:27.222328    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9" event={"ID":"3ad2b114-6a8e-4b73-8721-b9b11ce1b23e","Type":"ContainerDied","Data":"0c1e71c8aa60eaa60262cdce21a4c550650f45a85125b9a1e5f3805e7f19319c"}
Sep 06 19:56:27.222557 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:27.222350    2571 scope.go:117] "RemoveContainer" containerID="0366d7052cd6a88155d86cdc1d2dc7ae5049ac0f600d93e16814683c6b7f4d02"
Sep 06 19:56:27.231746 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:27.231725    2571 scope.go:117] "RemoveContainer" containerID="0366d7052cd6a88155d86cdc1d2dc7ae5049ac0f600d93e16814683c6b7f4d02"
Sep 06 19:56:27.232068 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:56:27.232049    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0366d7052cd6a88155d86cdc1d2dc7ae5049ac0f600d93e16814683c6b7f4d02\": container with ID starting with 0366d7052cd6a88155d86cdc1d2dc7ae5049ac0f600d93e16814683c6b7f4d02 not found: ID does not exist" containerID="0366d7052cd6a88155d86cdc1d2dc7ae5049ac0f600d93e16814683c6b7f4d02"
Sep 06 19:56:27.232125 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:27.232077    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0366d7052cd6a88155d86cdc1d2dc7ae5049ac0f600d93e16814683c6b7f4d02"} err="failed to get container status \"0366d7052cd6a88155d86cdc1d2dc7ae5049ac0f600d93e16814683c6b7f4d02\": rpc error: code = NotFound desc = could not find container \"0366d7052cd6a88155d86cdc1d2dc7ae5049ac0f600d93e16814683c6b7f4d02\": container with ID starting with 0366d7052cd6a88155d86cdc1d2dc7ae5049ac0f600d93e16814683c6b7f4d02 not found: ID does not exist"
Sep 06 19:56:27.239251 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:27.239224    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"]
Sep 06 19:56:27.241128 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:27.241107    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c95664-58896bf89d-9rqg9"]
Sep 06 19:56:28.924998 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:28.924943    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ad2b114-6a8e-4b73-8721-b9b11ce1b23e" path="/var/lib/kubelet/pods/3ad2b114-6a8e-4b73-8721-b9b11ce1b23e/volumes"
Sep 06 19:56:32.038104 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:32.038064    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:56:33.119688 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:33.119654    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"]
Sep 06 19:56:33.120163 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:33.119887    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt" podUID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerName="istio-proxy" containerID="cri-o://7fd19e65a5dbcc9a8ab1d2f89aafd72ff9250b7b61ffce2e88e6add396f89382" gracePeriod=30
Sep 06 19:56:36.471301 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:36.471251    2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 06 19:56:36.471713 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:36.471334    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt" podUID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 19:56:38.280634 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.280604    2571 generic.go:356] "Generic (PLEG): container finished" podID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerID="7fd19e65a5dbcc9a8ab1d2f89aafd72ff9250b7b61ffce2e88e6add396f89382" exitCode=0
Sep 06 19:56:38.281067 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.280674    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt" event={"ID":"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc","Type":"ContainerDied","Data":"7fd19e65a5dbcc9a8ab1d2f89aafd72ff9250b7b61ffce2e88e6add396f89382"}
Sep 06 19:56:38.281067 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.280722    2571 scope.go:117] "RemoveContainer" containerID="6fa5739f5a0bfa55c4ed1f4a47d286a613bcde5af0f43281ce5d2ea0cae80260"
Sep 06 19:56:38.379656 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.379629    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:56:38.494320 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.494213    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-podinfo\") pod \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") "
Sep 06 19:56:38.494320 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.494286    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-envoy\") pod \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") "
Sep 06 19:56:38.494320 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.494317    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istiod-ca-cert\") pod \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") "
Sep 06 19:56:38.494619 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.494349    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rcdp\" (UniqueName: \"kubernetes.io/projected/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-kube-api-access-9rcdp\") pod \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") "
Sep 06 19:56:38.494619 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.494386    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-workload-certs\") pod \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") "
Sep 06 19:56:38.494619 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.494410    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-data\") pod \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") "
Sep 06 19:56:38.494619 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.494481    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-credential-socket\") pod \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") "
Sep 06 19:56:38.494619 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.494505    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-token\") pod \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") "
Sep 06 19:56:38.494619 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.494538    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-workload-socket\") pod \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\" (UID: \"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc\") "
Sep 06 19:56:38.494915 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.494682    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" (UID: "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:56:38.494915 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.494698    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" (UID: "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:56:38.494915 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.494734    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" (UID: "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:56:38.494915 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.494894    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" (UID: "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:56:38.495153 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.494943    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" (UID: "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:56:38.495153 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.495065    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istiod-ca-cert\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:56:38.495153 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.495087    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-workload-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:56:38.495153 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.495099    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-data\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:56:38.495153 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.495111    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-credential-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:56:38.495153 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.495123    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-workload-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:56:38.496618 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.496594    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" (UID: "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 19:56:38.496709 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.496611    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-kube-api-access-9rcdp" (OuterVolumeSpecName: "kube-api-access-9rcdp") pod "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" (UID: "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc"). InnerVolumeSpecName "kube-api-access-9rcdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:56:38.496894 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.496880    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" (UID: "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:56:38.496977 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.496935    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" (UID: "8e3c7e71-38c7-48d2-8946-6a27aa5e14bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:56:38.596561 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.596526    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-podinfo\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:56:38.596561 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.596557    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-envoy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:56:38.596561 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.596566    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rcdp\" (UniqueName: \"kubernetes.io/projected/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-kube-api-access-9rcdp\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:56:38.596802 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:38.596575    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc-istio-token\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:56:39.287434 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:39.287333    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"
Sep 06 19:56:39.287434 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:39.287348    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt" event={"ID":"8e3c7e71-38c7-48d2-8946-6a27aa5e14bc","Type":"ContainerDied","Data":"cebc1014f96311bfc7e8a98875f4b686e64073d89692a8fce23773afcea71d48"}
Sep 06 19:56:39.287434 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:39.287432    2571 scope.go:117] "RemoveContainer" containerID="7fd19e65a5dbcc9a8ab1d2f89aafd72ff9250b7b61ffce2e88e6add396f89382"
Sep 06 19:56:39.304318 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:39.304287    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"]
Sep 06 19:56:39.306110 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:39.306088    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c95664-openshift-default-548cdb4dfd-mg7jt"]
Sep 06 19:56:40.917945 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:40.917914    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" path="/var/lib/kubelet/pods/8e3c7e71-38c7-48d2-8946-6a27aa5e14bc/volumes"
Sep 06 19:56:52.844355 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:52.844313    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:56:57.612892 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:56:57.612849    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"]
Sep 06 19:57:03.347186 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:03.347147    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:57:12.632726 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:12.632608    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2" podUID="3883d3ad-5748-4d31-8b92-7ef374ba5da4" containerName="main" containerID="cri-o://f110154ff05bb92077722f1419aed757cb7bef004cbaf7bba86fed0731e5152c" gracePeriod=45
Sep 06 19:57:12.895582 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:12.895513    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:57:13.047655 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.047623    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-dshm\") pod \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") "
Sep 06 19:57:13.047860 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.047687    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3883d3ad-5748-4d31-8b92-7ef374ba5da4-tls-certs\") pod \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") "
Sep 06 19:57:13.047860 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.047761    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-model-cache\") pod \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") "
Sep 06 19:57:13.047860 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.047795    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94vm6\" (UniqueName: \"kubernetes.io/projected/3883d3ad-5748-4d31-8b92-7ef374ba5da4-kube-api-access-94vm6\") pod \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") "
Sep 06 19:57:13.047860 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.047833    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-tmp-dir\") pod \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") "
Sep 06 19:57:13.048122 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.047887    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-home\") pod \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\" (UID: \"3883d3ad-5748-4d31-8b92-7ef374ba5da4\") "
Sep 06 19:57:13.048532 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.048175    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3883d3ad-5748-4d31-8b92-7ef374ba5da4" (UID: "3883d3ad-5748-4d31-8b92-7ef374ba5da4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:57:13.048532 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.048270    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-tmp-dir\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:57:13.048532 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.048339    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-home" (OuterVolumeSpecName: "home") pod "3883d3ad-5748-4d31-8b92-7ef374ba5da4" (UID: "3883d3ad-5748-4d31-8b92-7ef374ba5da4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:57:13.048532 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.048420    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-model-cache" (OuterVolumeSpecName: "model-cache") pod "3883d3ad-5748-4d31-8b92-7ef374ba5da4" (UID: "3883d3ad-5748-4d31-8b92-7ef374ba5da4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:57:13.050374 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.050335    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3883d3ad-5748-4d31-8b92-7ef374ba5da4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3883d3ad-5748-4d31-8b92-7ef374ba5da4" (UID: "3883d3ad-5748-4d31-8b92-7ef374ba5da4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:57:13.050474 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.050377    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-dshm" (OuterVolumeSpecName: "dshm") pod "3883d3ad-5748-4d31-8b92-7ef374ba5da4" (UID: "3883d3ad-5748-4d31-8b92-7ef374ba5da4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:57:13.051860 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.051832    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3883d3ad-5748-4d31-8b92-7ef374ba5da4-kube-api-access-94vm6" (OuterVolumeSpecName: "kube-api-access-94vm6") pod "3883d3ad-5748-4d31-8b92-7ef374ba5da4" (UID: "3883d3ad-5748-4d31-8b92-7ef374ba5da4"). InnerVolumeSpecName "kube-api-access-94vm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:57:13.149771 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.149667    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-dshm\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:57:13.149771 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.149703    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3883d3ad-5748-4d31-8b92-7ef374ba5da4-tls-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:57:13.149771 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.149717    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-model-cache\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:57:13.149771 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.149729    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94vm6\" (UniqueName: \"kubernetes.io/projected/3883d3ad-5748-4d31-8b92-7ef374ba5da4-kube-api-access-94vm6\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:57:13.149771 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.149742    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3883d3ad-5748-4d31-8b92-7ef374ba5da4-home\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:57:13.461218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.461122    2571 generic.go:356] "Generic (PLEG): container finished" podID="3883d3ad-5748-4d31-8b92-7ef374ba5da4" containerID="f110154ff05bb92077722f1419aed757cb7bef004cbaf7bba86fed0731e5152c" exitCode=0
Sep 06 19:57:13.461218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.461166    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"
Sep 06 19:57:13.461218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.461184    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2" event={"ID":"3883d3ad-5748-4d31-8b92-7ef374ba5da4","Type":"ContainerDied","Data":"f110154ff05bb92077722f1419aed757cb7bef004cbaf7bba86fed0731e5152c"}
Sep 06 19:57:13.461495 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.461224    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2" event={"ID":"3883d3ad-5748-4d31-8b92-7ef374ba5da4","Type":"ContainerDied","Data":"9e101c678492a62effd91c6b05b0f26c958169876c692ba4d283f9227184daff"}
Sep 06 19:57:13.461495 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.461246    2571 scope.go:117] "RemoveContainer" containerID="f110154ff05bb92077722f1419aed757cb7bef004cbaf7bba86fed0731e5152c"
Sep 06 19:57:13.470480 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.470455    2571 scope.go:117] "RemoveContainer" containerID="f110154ff05bb92077722f1419aed757cb7bef004cbaf7bba86fed0731e5152c"
Sep 06 19:57:13.470759 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:57:13.470739    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f110154ff05bb92077722f1419aed757cb7bef004cbaf7bba86fed0731e5152c\": container with ID starting with f110154ff05bb92077722f1419aed757cb7bef004cbaf7bba86fed0731e5152c not found: ID does not exist" containerID="f110154ff05bb92077722f1419aed757cb7bef004cbaf7bba86fed0731e5152c"
Sep 06 19:57:13.470815 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.470768    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f110154ff05bb92077722f1419aed757cb7bef004cbaf7bba86fed0731e5152c"} err="failed to get container status \"f110154ff05bb92077722f1419aed757cb7bef004cbaf7bba86fed0731e5152c\": rpc error: code = NotFound desc = could not find container \"f110154ff05bb92077722f1419aed757cb7bef004cbaf7bba86fed0731e5152c\": container with ID starting with f110154ff05bb92077722f1419aed757cb7bef004cbaf7bba86fed0731e5152c not found: ID does not exist"
Sep 06 19:57:13.481461 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.481440    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"]
Sep 06 19:57:13.485077 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:13.485052    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a30d6c/ipp-route-a30d6c-kserve-c8cfbbb75-f2nr2"]
Sep 06 19:57:14.918546 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:14.918508    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3883d3ad-5748-4d31-8b92-7ef374ba5da4" path="/var/lib/kubelet/pods/3883d3ad-5748-4d31-8b92-7ef374ba5da4/volumes"
Sep 06 19:57:17.027689 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:17.027660    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 19:57:17.031434 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:17.031411    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 19:57:51.103726 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.103686    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"]
Sep 06 19:57:51.104317 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.103939    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn" podUID="24b2b07e-3f36-4c55-9b4d-a115c816aaa3" containerName="maas-api" containerID="cri-o://0d5dd920c7905de1a5bc82bd39f99e59758039f9be10b57dd0d8e16f0f1983d9" gracePeriod=30
Sep 06 19:57:51.361772 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.361710    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:57:51.483354 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.483326    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfc28\" (UniqueName: \"kubernetes.io/projected/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-kube-api-access-pfc28\") pod \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") "
Sep 06 19:57:51.483537 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.483371    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-openshift-service-ca\") pod \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") "
Sep 06 19:57:51.483537 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.483392    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-maas-api-tls\") pod \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") "
Sep 06 19:57:51.483537 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.483519    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-metrics-certs\") pod \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\" (UID: \"24b2b07e-3f36-4c55-9b4d-a115c816aaa3\") "
Sep 06 19:57:51.483816 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.483788    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24b2b07e-3f36-4c55-9b4d-a115c816aaa3" (UID: "24b2b07e-3f36-4c55-9b4d-a115c816aaa3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:57:51.483975 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.483941    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-openshift-service-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:57:51.485791 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.485750    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-kube-api-access-pfc28" (OuterVolumeSpecName: "kube-api-access-pfc28") pod "24b2b07e-3f36-4c55-9b4d-a115c816aaa3" (UID: "24b2b07e-3f36-4c55-9b4d-a115c816aaa3"). InnerVolumeSpecName "kube-api-access-pfc28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:57:51.486068 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.486039    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "24b2b07e-3f36-4c55-9b4d-a115c816aaa3" (UID: "24b2b07e-3f36-4c55-9b4d-a115c816aaa3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:57:51.486183 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.486063    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24b2b07e-3f36-4c55-9b4d-a115c816aaa3" (UID: "24b2b07e-3f36-4c55-9b4d-a115c816aaa3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 19:57:51.585108 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.585057    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfc28\" (UniqueName: \"kubernetes.io/projected/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-kube-api-access-pfc28\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:57:51.585108 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.585103    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-maas-api-tls\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:57:51.585108 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.585113    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24b2b07e-3f36-4c55-9b4d-a115c816aaa3-metrics-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:57:51.656202 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.656116    2571 generic.go:356] "Generic (PLEG): container finished" podID="24b2b07e-3f36-4c55-9b4d-a115c816aaa3" containerID="0d5dd920c7905de1a5bc82bd39f99e59758039f9be10b57dd0d8e16f0f1983d9" exitCode=0
Sep 06 19:57:51.656202 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.656165    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"
Sep 06 19:57:51.656202 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.656180    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn" event={"ID":"24b2b07e-3f36-4c55-9b4d-a115c816aaa3","Type":"ContainerDied","Data":"0d5dd920c7905de1a5bc82bd39f99e59758039f9be10b57dd0d8e16f0f1983d9"}
Sep 06 19:57:51.656421 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.656222    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn" event={"ID":"24b2b07e-3f36-4c55-9b4d-a115c816aaa3","Type":"ContainerDied","Data":"ab35f95ff3a98ece7bcee694a37f089161926db448b7da76cba11ec54ff03acd"}
Sep 06 19:57:51.656421 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.656243    2571 scope.go:117] "RemoveContainer" containerID="0d5dd920c7905de1a5bc82bd39f99e59758039f9be10b57dd0d8e16f0f1983d9"
Sep 06 19:57:51.665996 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.665972    2571 scope.go:117] "RemoveContainer" containerID="0d5dd920c7905de1a5bc82bd39f99e59758039f9be10b57dd0d8e16f0f1983d9"
Sep 06 19:57:51.666309 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:57:51.666290    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5dd920c7905de1a5bc82bd39f99e59758039f9be10b57dd0d8e16f0f1983d9\": container with ID starting with 0d5dd920c7905de1a5bc82bd39f99e59758039f9be10b57dd0d8e16f0f1983d9 not found: ID does not exist" containerID="0d5dd920c7905de1a5bc82bd39f99e59758039f9be10b57dd0d8e16f0f1983d9"
Sep 06 19:57:51.666378 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.666320    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5dd920c7905de1a5bc82bd39f99e59758039f9be10b57dd0d8e16f0f1983d9"} err="failed to get container status \"0d5dd920c7905de1a5bc82bd39f99e59758039f9be10b57dd0d8e16f0f1983d9\": rpc error: code = NotFound desc = could not find container \"0d5dd920c7905de1a5bc82bd39f99e59758039f9be10b57dd0d8e16f0f1983d9\": container with ID starting with 0d5dd920c7905de1a5bc82bd39f99e59758039f9be10b57dd0d8e16f0f1983d9 not found: ID does not exist"
Sep 06 19:57:51.680232 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.680206    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"]
Sep 06 19:57:51.684088 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:51.684064    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-adaf11-5d8b79b8f5-nhnhn"]
Sep 06 19:57:52.918433 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:52.918397    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24b2b07e-3f36-4c55-9b4d-a115c816aaa3" path="/var/lib/kubelet/pods/24b2b07e-3f36-4c55-9b4d-a115c816aaa3/volumes"
Sep 06 19:57:56.041708 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:56.041677    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:57:57.541478 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.541442    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"]
Sep 06 19:57:57.542099 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.541681    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq" podUID="e8ad7d2f-e606-4275-981f-31f9ff66c8e7" containerName="istio-proxy" containerID="cri-o://4c7396cc3005c45723bc8cafa87d29f3cf2fa1ee9260074e3a0af1a16b9b18f4" gracePeriod=30
Sep 06 19:57:57.882577 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.882522    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt"]
Sep 06 19:57:57.883050 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.883030    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerName="istio-proxy"
Sep 06 19:57:57.883050 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.883050    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerName="istio-proxy"
Sep 06 19:57:57.883230 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.883063    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b2b07e-3f36-4c55-9b4d-a115c816aaa3" containerName="maas-api"
Sep 06 19:57:57.883230 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.883072    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b2b07e-3f36-4c55-9b4d-a115c816aaa3" containerName="maas-api"
Sep 06 19:57:57.883230 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.883084    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ad2b114-6a8e-4b73-8721-b9b11ce1b23e" containerName="maas-api"
Sep 06 19:57:57.883230 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.883092    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ad2b114-6a8e-4b73-8721-b9b11ce1b23e" containerName="maas-api"
Sep 06 19:57:57.883230 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.883104    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3883d3ad-5748-4d31-8b92-7ef374ba5da4" containerName="main"
Sep 06 19:57:57.883230 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.883111    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3883d3ad-5748-4d31-8b92-7ef374ba5da4" containerName="main"
Sep 06 19:57:57.883230 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.883222    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerName="istio-proxy"
Sep 06 19:57:57.883230 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.883233    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3883d3ad-5748-4d31-8b92-7ef374ba5da4" containerName="main"
Sep 06 19:57:57.883611 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.883246    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="24b2b07e-3f36-4c55-9b4d-a115c816aaa3" containerName="maas-api"
Sep 06 19:57:57.883611 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.883257    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ad2b114-6a8e-4b73-8721-b9b11ce1b23e" containerName="maas-api"
Sep 06 19:57:57.887791 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.887760    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt"
Sep 06 19:57:57.890313 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.890291    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 19:57:57.890442 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.890399    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h45ww\""
Sep 06 19:57:57.892203 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.892174    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt"]
Sep 06 19:57:57.948418 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:57.948378    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1908d467-2ce1-4283-99e8-4c3420597d9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt\" (UID: \"1908d467-2ce1-4283-99e8-4c3420597d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt"
Sep 06 19:57:58.049803 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:58.049762    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1908d467-2ce1-4283-99e8-4c3420597d9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt\" (UID: \"1908d467-2ce1-4283-99e8-4c3420597d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt"
Sep 06 19:57:58.050490 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:58.050470    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1908d467-2ce1-4283-99e8-4c3420597d9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt\" (UID: \"1908d467-2ce1-4283-99e8-4c3420597d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt"
Sep 06 19:57:58.200358 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:58.200256    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt"
Sep 06 19:57:58.468339 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:58.468313    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt"]
Sep 06 19:57:58.471136 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:57:58.471098    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1908d467_2ce1_4283_99e8_4c3420597d9b.slice/crio-1890ca3c37615bf638afbf4cc71daa89b1dae84420441b233e441ba942e3f8fc WatchSource:0}: Error finding container 1890ca3c37615bf638afbf4cc71daa89b1dae84420441b233e441ba942e3f8fc: Status 404 returned error can't find the container with id 1890ca3c37615bf638afbf4cc71daa89b1dae84420441b233e441ba942e3f8fc
Sep 06 19:57:58.693428 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:58.693398    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt" event={"ID":"1908d467-2ce1-4283-99e8-4c3420597d9b","Type":"ContainerStarted","Data":"1890ca3c37615bf638afbf4cc71daa89b1dae84420441b233e441ba942e3f8fc"}
Sep 06 19:57:59.700534 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:59.700496    2571 generic.go:356] "Generic (PLEG): container finished" podID="1908d467-2ce1-4283-99e8-4c3420597d9b" containerID="fe4f4ae93df65b5c0d7707f25424b060f3f9612f402f74e2e46c4b680420f028" exitCode=0
Sep 06 19:57:59.700923 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:57:59.700580    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt" event={"ID":"1908d467-2ce1-4283-99e8-4c3420597d9b","Type":"ContainerDied","Data":"fe4f4ae93df65b5c0d7707f25424b060f3f9612f402f74e2e46c4b680420f028"}
Sep 06 19:58:00.835795 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:00.835761    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt"
Sep 06 19:58:00.978469 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:00.978388    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1908d467-2ce1-4283-99e8-4c3420597d9b-maas-api-service-ca\") pod \"1908d467-2ce1-4283-99e8-4c3420597d9b\" (UID: \"1908d467-2ce1-4283-99e8-4c3420597d9b\") "
Sep 06 19:58:00.978731 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:00.978707    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1908d467-2ce1-4283-99e8-4c3420597d9b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1908d467-2ce1-4283-99e8-4c3420597d9b" (UID: "1908d467-2ce1-4283-99e8-4c3420597d9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:58:01.080058 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:01.080020    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1908d467-2ce1-4283-99e8-4c3420597d9b-maas-api-service-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:01.712047 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:01.712003    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt" event={"ID":"1908d467-2ce1-4283-99e8-4c3420597d9b","Type":"ContainerDied","Data":"1890ca3c37615bf638afbf4cc71daa89b1dae84420441b233e441ba942e3f8fc"}
Sep 06 19:58:01.712229 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:01.712061    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1890ca3c37615bf638afbf4cc71daa89b1dae84420441b233e441ba942e3f8fc"
Sep 06 19:58:01.712229 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:01.712020    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt"
Sep 06 19:58:02.720161 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.720128    2571 generic.go:356] "Generic (PLEG): container finished" podID="e8ad7d2f-e606-4275-981f-31f9ff66c8e7" containerID="4c7396cc3005c45723bc8cafa87d29f3cf2fa1ee9260074e3a0af1a16b9b18f4" exitCode=0
Sep 06 19:58:02.720548 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.720205    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq" event={"ID":"e8ad7d2f-e606-4275-981f-31f9ff66c8e7","Type":"ContainerDied","Data":"4c7396cc3005c45723bc8cafa87d29f3cf2fa1ee9260074e3a0af1a16b9b18f4"}
Sep 06 19:58:02.720548 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.720281    2571 scope.go:117] "RemoveContainer" containerID="79223ab9010b3ed3bdfa61d68e164ccb3f43aab6500fe604c40bc86468b2ee6c"
Sep 06 19:58:02.801918 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.801893    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:58:02.898035 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898005    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-podinfo\") pod \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") "
Sep 06 19:58:02.898220 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898057    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-workload-certs\") pod \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") "
Sep 06 19:58:02.898220 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898101    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-data\") pod \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") "
Sep 06 19:58:02.898220 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898121    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-token\") pod \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") "
Sep 06 19:58:02.898220 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898161    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istiod-ca-cert\") pod \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") "
Sep 06 19:58:02.898220 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898199    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-credential-socket\") pod \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") "
Sep 06 19:58:02.898486 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898260    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-workload-socket\") pod \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") "
Sep 06 19:58:02.898486 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898293    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-envoy\") pod \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") "
Sep 06 19:58:02.898486 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898331    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfbb2\" (UniqueName: \"kubernetes.io/projected/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-kube-api-access-kfbb2\") pod \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\" (UID: \"e8ad7d2f-e606-4275-981f-31f9ff66c8e7\") "
Sep 06 19:58:02.898644 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898483    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e8ad7d2f-e606-4275-981f-31f9ff66c8e7" (UID: "e8ad7d2f-e606-4275-981f-31f9ff66c8e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:58:02.898644 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898521    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "e8ad7d2f-e606-4275-981f-31f9ff66c8e7" (UID: "e8ad7d2f-e606-4275-981f-31f9ff66c8e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:58:02.898644 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898559    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e8ad7d2f-e606-4275-981f-31f9ff66c8e7" (UID: "e8ad7d2f-e606-4275-981f-31f9ff66c8e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:58:02.898787 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898680    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e8ad7d2f-e606-4275-981f-31f9ff66c8e7" (UID: "e8ad7d2f-e606-4275-981f-31f9ff66c8e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:58:02.898787 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898745    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e8ad7d2f-e606-4275-981f-31f9ff66c8e7" (UID: "e8ad7d2f-e606-4275-981f-31f9ff66c8e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:58:02.898787 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898765    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-workload-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:02.898787 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898783    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-workload-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:02.898932 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898795    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-data\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:02.898932 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.898804    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-credential-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:02.900553 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.900528    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-kube-api-access-kfbb2" (OuterVolumeSpecName: "kube-api-access-kfbb2") pod "e8ad7d2f-e606-4275-981f-31f9ff66c8e7" (UID: "e8ad7d2f-e606-4275-981f-31f9ff66c8e7"). InnerVolumeSpecName "kube-api-access-kfbb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:58:02.900660 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.900604    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e8ad7d2f-e606-4275-981f-31f9ff66c8e7" (UID: "e8ad7d2f-e606-4275-981f-31f9ff66c8e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:58:02.900798 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.900780    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e8ad7d2f-e606-4275-981f-31f9ff66c8e7" (UID: "e8ad7d2f-e606-4275-981f-31f9ff66c8e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 19:58:02.900872 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.900793    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "e8ad7d2f-e606-4275-981f-31f9ff66c8e7" (UID: "e8ad7d2f-e606-4275-981f-31f9ff66c8e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:58:02.999485 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.999444    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-podinfo\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:02.999485 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.999479    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-token\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:02.999485 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.999492    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istiod-ca-cert\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:02.999732 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.999502    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-istio-envoy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:02.999732 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:02.999511    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfbb2\" (UniqueName: \"kubernetes.io/projected/e8ad7d2f-e606-4275-981f-31f9ff66c8e7-kube-api-access-kfbb2\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:03.726925 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:03.726896    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"
Sep 06 19:58:03.726925 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:03.726909    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq" event={"ID":"e8ad7d2f-e606-4275-981f-31f9ff66c8e7","Type":"ContainerDied","Data":"d4948508ac519522a6077ce0ab6d3742a41951123f869dff0c18aab5f553dcd7"}
Sep 06 19:58:03.727442 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:03.726980    2571 scope.go:117] "RemoveContainer" containerID="4c7396cc3005c45723bc8cafa87d29f3cf2fa1ee9260074e3a0af1a16b9b18f4"
Sep 06 19:58:03.743923 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:03.743898    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"]
Sep 06 19:58:03.749858 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:03.749821    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-adaf11-openshift-default-867cfdd788-rpwqq"]
Sep 06 19:58:04.918218 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:04.918180    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8ad7d2f-e606-4275-981f-31f9ff66c8e7" path="/var/lib/kubelet/pods/e8ad7d2f-e606-4275-981f-31f9ff66c8e7/volumes"
Sep 06 19:58:08.617124 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:08.617086    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt"]
Sep 06 19:58:08.618823 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:08.618799    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b571c4-wnwxt"]
Sep 06 19:58:08.918360 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:08.918258    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1908d467-2ce1-4283-99e8-4c3420597d9b" path="/var/lib/kubelet/pods/1908d467-2ce1-4283-99e8-4c3420597d9b/volumes"
Sep 06 19:58:10.734199 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:10.734159    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:58:15.765850 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.765811    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v"]
Sep 06 19:58:15.766264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.766180    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ad7d2f-e606-4275-981f-31f9ff66c8e7" containerName="istio-proxy"
Sep 06 19:58:15.766264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.766192    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ad7d2f-e606-4275-981f-31f9ff66c8e7" containerName="istio-proxy"
Sep 06 19:58:15.766264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.766207    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ad7d2f-e606-4275-981f-31f9ff66c8e7" containerName="istio-proxy"
Sep 06 19:58:15.766264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.766213    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ad7d2f-e606-4275-981f-31f9ff66c8e7" containerName="istio-proxy"
Sep 06 19:58:15.766264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.766221    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerName="istio-proxy"
Sep 06 19:58:15.766264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.766227    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerName="istio-proxy"
Sep 06 19:58:15.766264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.766243    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1908d467-2ce1-4283-99e8-4c3420597d9b" containerName="revoke-keys"
Sep 06 19:58:15.766264 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.766248    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1908d467-2ce1-4283-99e8-4c3420597d9b" containerName="revoke-keys"
Sep 06 19:58:15.766558 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.766301    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ad7d2f-e606-4275-981f-31f9ff66c8e7" containerName="istio-proxy"
Sep 06 19:58:15.766558 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.766309    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ad7d2f-e606-4275-981f-31f9ff66c8e7" containerName="istio-proxy"
Sep 06 19:58:15.766558 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.766316    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1908d467-2ce1-4283-99e8-4c3420597d9b" containerName="revoke-keys"
Sep 06 19:58:15.766558 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.766321    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e3c7e71-38c7-48d2-8946-6a27aa5e14bc" containerName="istio-proxy"
Sep 06 19:58:15.770834 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.770816    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v"
Sep 06 19:58:15.773626 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.773597    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 19:58:15.773765 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.773597    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h45ww\""
Sep 06 19:58:15.775334 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.775306    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v"]
Sep 06 19:58:15.823206 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.823170    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a161e2-366d-4cce-85b9-3595777973bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v\" (UID: \"d8a161e2-366d-4cce-85b9-3595777973bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v"
Sep 06 19:58:15.924586 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.924547    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a161e2-366d-4cce-85b9-3595777973bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v\" (UID: \"d8a161e2-366d-4cce-85b9-3595777973bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v"
Sep 06 19:58:15.925295 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:15.925266    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a161e2-366d-4cce-85b9-3595777973bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v\" (UID: \"d8a161e2-366d-4cce-85b9-3595777973bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v"
Sep 06 19:58:16.083423 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:16.083395    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v"
Sep 06 19:58:16.354434 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:16.354410    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v"]
Sep 06 19:58:16.356381 ip-10-0-141-42 kubenswrapper[2571]: W0906 19:58:16.356350    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a161e2_366d_4cce_85b9_3595777973bd.slice/crio-779f84f8abd74934e75ce4f0e1f99faa8c6bfb69086d77793a069be7e7a8279c WatchSource:0}: Error finding container 779f84f8abd74934e75ce4f0e1f99faa8c6bfb69086d77793a069be7e7a8279c: Status 404 returned error can't find the container with id 779f84f8abd74934e75ce4f0e1f99faa8c6bfb69086d77793a069be7e7a8279c
Sep 06 19:58:16.798682 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:16.798585    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v" event={"ID":"d8a161e2-366d-4cce-85b9-3595777973bd","Type":"ContainerStarted","Data":"d5d7a2429b3a65d8b000719632245862bddd42c27915e4d91c490551d07d26a2"}
Sep 06 19:58:16.798682 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:16.798637    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v" event={"ID":"d8a161e2-366d-4cce-85b9-3595777973bd","Type":"ContainerStarted","Data":"779f84f8abd74934e75ce4f0e1f99faa8c6bfb69086d77793a069be7e7a8279c"}
Sep 06 19:58:16.812674 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:16.812624    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v" podStartSLOduration=1.8126094529999999 podStartE2EDuration="1.812609453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 19:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 19:58:16.811187269 +0000 UTC m=+1860.381485616" watchObservedRunningTime="2026-09-06 19:58:16.812609453 +0000 UTC m=+1860.382907800"
Sep 06 19:58:17.348372 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:17.348341    2571 scope.go:117] "RemoveContainer" containerID="26a79a4cecf9861e128364b9357e9f5f6446280fe34faf9ff100eff0e7b94127"
Sep 06 19:58:17.357170 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:17.357144    2571 scope.go:117] "RemoveContainer" containerID="2615a4dd58fa758c4ac9e906368f956be019550aaebae85ce846d2f9663fffcd"
Sep 06 19:58:17.805025 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:17.804923    2571 generic.go:356] "Generic (PLEG): container finished" podID="d8a161e2-366d-4cce-85b9-3595777973bd" containerID="d5d7a2429b3a65d8b000719632245862bddd42c27915e4d91c490551d07d26a2" exitCode=0
Sep 06 19:58:17.805025 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:17.805005    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v" event={"ID":"d8a161e2-366d-4cce-85b9-3595777973bd","Type":"ContainerDied","Data":"d5d7a2429b3a65d8b000719632245862bddd42c27915e4d91c490551d07d26a2"}
Sep 06 19:58:18.947330 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:18.947298    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v"
Sep 06 19:58:19.055090 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:19.055049    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a161e2-366d-4cce-85b9-3595777973bd-maas-api-service-ca\") pod \"d8a161e2-366d-4cce-85b9-3595777973bd\" (UID: \"d8a161e2-366d-4cce-85b9-3595777973bd\") "
Sep 06 19:58:19.055478 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:19.055456    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a161e2-366d-4cce-85b9-3595777973bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8a161e2-366d-4cce-85b9-3595777973bd" (UID: "d8a161e2-366d-4cce-85b9-3595777973bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:58:19.156778 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:19.156732    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a161e2-366d-4cce-85b9-3595777973bd-maas-api-service-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:19.815461 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:19.815426    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v"
Sep 06 19:58:19.815647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:19.815456    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v" event={"ID":"d8a161e2-366d-4cce-85b9-3595777973bd","Type":"ContainerDied","Data":"779f84f8abd74934e75ce4f0e1f99faa8c6bfb69086d77793a069be7e7a8279c"}
Sep 06 19:58:19.815647 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:19.815534    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="779f84f8abd74934e75ce4f0e1f99faa8c6bfb69086d77793a069be7e7a8279c"
Sep 06 19:58:26.344040 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:26.344002    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v"]
Sep 06 19:58:26.346561 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:26.346535    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-712616-s499v"]
Sep 06 19:58:26.924009 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:26.923973    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8a161e2-366d-4cce-85b9-3595777973bd" path="/var/lib/kubelet/pods/d8a161e2-366d-4cce-85b9-3595777973bd/volumes"
Sep 06 19:58:28.841230 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:28.841192    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:58:32.584526 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:32.584487    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"]
Sep 06 19:58:32.585007 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:32.584749    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6" podUID="066db7cd-7ba1-49d0-8539-2f6cde582dae" containerName="istio-proxy" containerID="cri-o://1afc58af76fe5a3422a012941b51edfd8f98177f658d7363b75ac9c200c2bae1" gracePeriod=30
Sep 06 19:58:37.237558 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.237525    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 19:58:37.835082 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.835054    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:58:37.910493 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.910400    2571 generic.go:356] "Generic (PLEG): container finished" podID="066db7cd-7ba1-49d0-8539-2f6cde582dae" containerID="1afc58af76fe5a3422a012941b51edfd8f98177f658d7363b75ac9c200c2bae1" exitCode=0
Sep 06 19:58:37.910493 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.910459    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"
Sep 06 19:58:37.910754 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.910486    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6" event={"ID":"066db7cd-7ba1-49d0-8539-2f6cde582dae","Type":"ContainerDied","Data":"1afc58af76fe5a3422a012941b51edfd8f98177f658d7363b75ac9c200c2bae1"}
Sep 06 19:58:37.910754 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.910545    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6" event={"ID":"066db7cd-7ba1-49d0-8539-2f6cde582dae","Type":"ContainerDied","Data":"1eef30594bdabdca148795a85c3ff739d036e420e9dc02ea064078d8bc3ed2a1"}
Sep 06 19:58:37.910754 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.910575    2571 scope.go:117] "RemoveContainer" containerID="1afc58af76fe5a3422a012941b51edfd8f98177f658d7363b75ac9c200c2bae1"
Sep 06 19:58:37.919824 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.919804    2571 scope.go:117] "RemoveContainer" containerID="222c93b303556e4b28665502add137dde1a7524559db37aa80d3fa12ffca58f5"
Sep 06 19:58:37.928767 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.928740    2571 scope.go:117] "RemoveContainer" containerID="1afc58af76fe5a3422a012941b51edfd8f98177f658d7363b75ac9c200c2bae1"
Sep 06 19:58:37.929087 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:58:37.929064    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1afc58af76fe5a3422a012941b51edfd8f98177f658d7363b75ac9c200c2bae1\": container with ID starting with 1afc58af76fe5a3422a012941b51edfd8f98177f658d7363b75ac9c200c2bae1 not found: ID does not exist" containerID="1afc58af76fe5a3422a012941b51edfd8f98177f658d7363b75ac9c200c2bae1"
Sep 06 19:58:37.929171 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.929095    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1afc58af76fe5a3422a012941b51edfd8f98177f658d7363b75ac9c200c2bae1"} err="failed to get container status \"1afc58af76fe5a3422a012941b51edfd8f98177f658d7363b75ac9c200c2bae1\": rpc error: code = NotFound desc = could not find container \"1afc58af76fe5a3422a012941b51edfd8f98177f658d7363b75ac9c200c2bae1\": container with ID starting with 1afc58af76fe5a3422a012941b51edfd8f98177f658d7363b75ac9c200c2bae1 not found: ID does not exist"
Sep 06 19:58:37.929171 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.929112    2571 scope.go:117] "RemoveContainer" containerID="222c93b303556e4b28665502add137dde1a7524559db37aa80d3fa12ffca58f5"
Sep 06 19:58:37.929377 ip-10-0-141-42 kubenswrapper[2571]: E0906 19:58:37.929357    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"222c93b303556e4b28665502add137dde1a7524559db37aa80d3fa12ffca58f5\": container with ID starting with 222c93b303556e4b28665502add137dde1a7524559db37aa80d3fa12ffca58f5 not found: ID does not exist" containerID="222c93b303556e4b28665502add137dde1a7524559db37aa80d3fa12ffca58f5"
Sep 06 19:58:37.929426 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.929385    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"222c93b303556e4b28665502add137dde1a7524559db37aa80d3fa12ffca58f5"} err="failed to get container status \"222c93b303556e4b28665502add137dde1a7524559db37aa80d3fa12ffca58f5\": rpc error: code = NotFound desc = could not find container \"222c93b303556e4b28665502add137dde1a7524559db37aa80d3fa12ffca58f5\": container with ID starting with 222c93b303556e4b28665502add137dde1a7524559db37aa80d3fa12ffca58f5 not found: ID does not exist"
Sep 06 19:58:37.940790 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.940765    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-podinfo\") pod \"066db7cd-7ba1-49d0-8539-2f6cde582dae\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") "
Sep 06 19:58:37.940912 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.940831    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-workload-certs\") pod \"066db7cd-7ba1-49d0-8539-2f6cde582dae\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") "
Sep 06 19:58:37.940912 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.940863    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-workload-socket\") pod \"066db7cd-7ba1-49d0-8539-2f6cde582dae\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") "
Sep 06 19:58:37.941009 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.940910    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-data\") pod \"066db7cd-7ba1-49d0-8539-2f6cde582dae\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") "
Sep 06 19:58:37.941009 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.940941    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-envoy\") pod \"066db7cd-7ba1-49d0-8539-2f6cde582dae\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") "
Sep 06 19:58:37.941009 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.940986    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-token\") pod \"066db7cd-7ba1-49d0-8539-2f6cde582dae\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") "
Sep 06 19:58:37.941155 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.941015    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-credential-socket\") pod \"066db7cd-7ba1-49d0-8539-2f6cde582dae\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") "
Sep 06 19:58:37.941155 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.941058    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cn48h\" (UniqueName: \"kubernetes.io/projected/066db7cd-7ba1-49d0-8539-2f6cde582dae-kube-api-access-cn48h\") pod \"066db7cd-7ba1-49d0-8539-2f6cde582dae\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") "
Sep 06 19:58:37.941155 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.941094    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066db7cd-7ba1-49d0-8539-2f6cde582dae-istiod-ca-cert\") pod \"066db7cd-7ba1-49d0-8539-2f6cde582dae\" (UID: \"066db7cd-7ba1-49d0-8539-2f6cde582dae\") "
Sep 06 19:58:37.941312 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.941181    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "066db7cd-7ba1-49d0-8539-2f6cde582dae" (UID: "066db7cd-7ba1-49d0-8539-2f6cde582dae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:58:37.941312 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.941202    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-data" (OuterVolumeSpecName: "istio-data") pod "066db7cd-7ba1-49d0-8539-2f6cde582dae" (UID: "066db7cd-7ba1-49d0-8539-2f6cde582dae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:58:37.941433 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.941398    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "066db7cd-7ba1-49d0-8539-2f6cde582dae" (UID: "066db7cd-7ba1-49d0-8539-2f6cde582dae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:58:37.941490 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.941468    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-workload-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:37.941550 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.941489    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-data\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:37.941762 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.941702    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "066db7cd-7ba1-49d0-8539-2f6cde582dae" (UID: "066db7cd-7ba1-49d0-8539-2f6cde582dae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:58:37.941843 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.941761    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066db7cd-7ba1-49d0-8539-2f6cde582dae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "066db7cd-7ba1-49d0-8539-2f6cde582dae" (UID: "066db7cd-7ba1-49d0-8539-2f6cde582dae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 19:58:37.943103 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.943078    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "066db7cd-7ba1-49d0-8539-2f6cde582dae" (UID: "066db7cd-7ba1-49d0-8539-2f6cde582dae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 19:58:37.943529 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.943499    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "066db7cd-7ba1-49d0-8539-2f6cde582dae" (UID: "066db7cd-7ba1-49d0-8539-2f6cde582dae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 19:58:37.943666 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.943646    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066db7cd-7ba1-49d0-8539-2f6cde582dae-kube-api-access-cn48h" (OuterVolumeSpecName: "kube-api-access-cn48h") pod "066db7cd-7ba1-49d0-8539-2f6cde582dae" (UID: "066db7cd-7ba1-49d0-8539-2f6cde582dae"). InnerVolumeSpecName "kube-api-access-cn48h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:58:37.943666 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:37.943640    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-token" (OuterVolumeSpecName: "istio-token") pod "066db7cd-7ba1-49d0-8539-2f6cde582dae" (UID: "066db7cd-7ba1-49d0-8539-2f6cde582dae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 19:58:38.042175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:38.042134    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-podinfo\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:38.042175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:38.042172    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-workload-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:38.042175 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:38.042184    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-envoy\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:38.042431 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:38.042196    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066db7cd-7ba1-49d0-8539-2f6cde582dae-istio-token\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:38.042431 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:38.042207    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066db7cd-7ba1-49d0-8539-2f6cde582dae-credential-socket\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:38.042431 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:38.042220    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn48h\" (UniqueName: \"kubernetes.io/projected/066db7cd-7ba1-49d0-8539-2f6cde582dae-kube-api-access-cn48h\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:38.042431 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:38.042231    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066db7cd-7ba1-49d0-8539-2f6cde582dae-istiod-ca-cert\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 19:58:38.232852 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:38.232775    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"]
Sep 06 19:58:38.234803 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:38.234774    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-712616-openshift-default-74d7599588-gznr6"]
Sep 06 19:58:38.924934 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:58:38.924846    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="066db7cd-7ba1-49d0-8539-2f6cde582dae" path="/var/lib/kubelet/pods/066db7cd-7ba1-49d0-8539-2f6cde582dae/volumes"
Sep 06 19:59:56.635423 ip-10-0-141-42 kubenswrapper[2571]: I0906 19:59:56.635379    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:00:19.933437 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:00:19.933348    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:00:26.641298 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:00:26.641264    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:00:34.026236 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:00:34.026195    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:00:49.530656 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:00:49.530619    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:00:55.437424 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:00:55.437385    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:02:17.058813 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:17.058781    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 20:02:17.064751 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:17.064720    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 20:02:27.344263 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:27.344223    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:02:33.241627 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:33.241587    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:02:43.047317 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.047281    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"]
Sep 06 20:02:43.047688 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.047630    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066db7cd-7ba1-49d0-8539-2f6cde582dae" containerName="istio-proxy"
Sep 06 20:02:43.047688 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.047642    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="066db7cd-7ba1-49d0-8539-2f6cde582dae" containerName="istio-proxy"
Sep 06 20:02:43.047688 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.047650    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066db7cd-7ba1-49d0-8539-2f6cde582dae" containerName="istio-proxy"
Sep 06 20:02:43.047688 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.047656    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="066db7cd-7ba1-49d0-8539-2f6cde582dae" containerName="istio-proxy"
Sep 06 20:02:43.047688 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.047672    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8a161e2-366d-4cce-85b9-3595777973bd" containerName="revoke-keys"
Sep 06 20:02:43.047688 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.047677    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8a161e2-366d-4cce-85b9-3595777973bd" containerName="revoke-keys"
Sep 06 20:02:43.047888 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.047766    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="066db7cd-7ba1-49d0-8539-2f6cde582dae" containerName="istio-proxy"
Sep 06 20:02:43.047888 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.047775    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="066db7cd-7ba1-49d0-8539-2f6cde582dae" containerName="istio-proxy"
Sep 06 20:02:43.047888 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.047784    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8a161e2-366d-4cce-85b9-3595777973bd" containerName="revoke-keys"
Sep 06 20:02:43.050850 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.050833    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.053453 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.053433    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-90c1ebd6-kserve-self-signed-certs\""
Sep 06 20:02:43.061637 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.061612    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"]
Sep 06 20:02:43.145825 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.145787    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-home\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.146070 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.145832    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-dshm\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.146070 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.145872    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-tmp-dir\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.146070 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.145947    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-tls-certs\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.146070 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.146004    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-model-cache\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.146070 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.146026    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6xfs\" (UniqueName: \"kubernetes.io/projected/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-kube-api-access-b6xfs\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.246488 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.246455    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-tls-certs\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.246658 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.246498    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-model-cache\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.246658 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.246524    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6xfs\" (UniqueName: \"kubernetes.io/projected/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-kube-api-access-b6xfs\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.246658 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.246584    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-home\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.246658 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.246617    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-dshm\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.246658 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.246642    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-tmp-dir\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.246934 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.246915    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-model-cache\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.247022 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.246978    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-home\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.247137 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.247112    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-tmp-dir\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.248908 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.248878    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-dshm\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.249027 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.248921    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-tls-certs\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.254111 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.254087    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6xfs\" (UniqueName: \"kubernetes.io/projected/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-kube-api-access-b6xfs\") pod \"e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") " pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.360039 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.360002    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:43.634600 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:43.634570    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"]
Sep 06 20:02:43.637009 ip-10-0-141-42 kubenswrapper[2571]: W0906 20:02:43.636981    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0e3abb4_4ab5_4b18_a497_12c812e80fb1.slice/crio-d620f477d79a709cc5c3400f9d9b897841548b95b2337a7b327f38233ef2c1e5 WatchSource:0}: Error finding container d620f477d79a709cc5c3400f9d9b897841548b95b2337a7b327f38233ef2c1e5: Status 404 returned error can't find the container with id d620f477d79a709cc5c3400f9d9b897841548b95b2337a7b327f38233ef2c1e5
Sep 06 20:02:44.120415 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:44.120382    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r" event={"ID":"d0e3abb4-4ab5-4b18-a497-12c812e80fb1","Type":"ContainerStarted","Data":"98bd321d32c021a66839b6759e665fd84934a1187aef0fed8a748917e8391a1f"}
Sep 06 20:02:44.120794 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:44.120425    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r" event={"ID":"d0e3abb4-4ab5-4b18-a497-12c812e80fb1","Type":"ContainerStarted","Data":"d620f477d79a709cc5c3400f9d9b897841548b95b2337a7b327f38233ef2c1e5"}
Sep 06 20:02:44.136369 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:44.136323    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r" podStartSLOduration=1.136309339 podStartE2EDuration="1.136309339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 20:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 20:02:44.134896595 +0000 UTC m=+2127.705194940" watchObservedRunningTime="2026-09-06 20:02:44.136309339 +0000 UTC m=+2127.706607679"
Sep 06 20:02:45.747267 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:45.747227    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:02:53.360193 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:53.360143    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:53.360675 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:53.360210    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:53.372828 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:53.372802    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:54.182324 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:54.182295    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:02:56.741629 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:02:56.741589    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:03:15.866773 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:15.866693    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"]
Sep 06 20:03:19.146813 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:19.146773    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:03:27.234477 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:27.234439    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:03:30.886819 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:30.886775    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r" podUID="d0e3abb4-4ab5-4b18-a497-12c812e80fb1" containerName="main" containerID="cri-o://98bd321d32c021a66839b6759e665fd84934a1187aef0fed8a748917e8391a1f" gracePeriod=45
Sep 06 20:03:31.140558 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.140496    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:03:31.192345 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.192312    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-dshm\") pod \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") "
Sep 06 20:03:31.192345 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.192349    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-home\") pod \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") "
Sep 06 20:03:31.192572 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.192373    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-model-cache\") pod \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") "
Sep 06 20:03:31.192572 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.192410    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6xfs\" (UniqueName: \"kubernetes.io/projected/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-kube-api-access-b6xfs\") pod \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") "
Sep 06 20:03:31.192572 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.192465    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-tmp-dir\") pod \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") "
Sep 06 20:03:31.192572 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.192494    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-tls-certs\") pod \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\" (UID: \"d0e3abb4-4ab5-4b18-a497-12c812e80fb1\") "
Sep 06 20:03:31.192791 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.192654    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-home" (OuterVolumeSpecName: "home") pod "d0e3abb4-4ab5-4b18-a497-12c812e80fb1" (UID: "d0e3abb4-4ab5-4b18-a497-12c812e80fb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 20:03:31.192791 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.192706    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0e3abb4-4ab5-4b18-a497-12c812e80fb1" (UID: "d0e3abb4-4ab5-4b18-a497-12c812e80fb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 20:03:31.192791 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.192714    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0e3abb4-4ab5-4b18-a497-12c812e80fb1" (UID: "d0e3abb4-4ab5-4b18-a497-12c812e80fb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 20:03:31.192926 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.192811    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-tmp-dir\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 20:03:31.192926 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.192827    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-home\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 20:03:31.192926 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.192836    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-model-cache\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 20:03:31.194590 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.194565    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-kube-api-access-b6xfs" (OuterVolumeSpecName: "kube-api-access-b6xfs") pod "d0e3abb4-4ab5-4b18-a497-12c812e80fb1" (UID: "d0e3abb4-4ab5-4b18-a497-12c812e80fb1"). InnerVolumeSpecName "kube-api-access-b6xfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 20:03:31.194699 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.194590    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0e3abb4-4ab5-4b18-a497-12c812e80fb1" (UID: "d0e3abb4-4ab5-4b18-a497-12c812e80fb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 20:03:31.194699 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.194593    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-dshm" (OuterVolumeSpecName: "dshm") pod "d0e3abb4-4ab5-4b18-a497-12c812e80fb1" (UID: "d0e3abb4-4ab5-4b18-a497-12c812e80fb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 20:03:31.293501 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.293458    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-tls-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 20:03:31.293501 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.293496    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-dshm\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 20:03:31.293501 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.293507    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6xfs\" (UniqueName: \"kubernetes.io/projected/d0e3abb4-4ab5-4b18-a497-12c812e80fb1-kube-api-access-b6xfs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 20:03:31.359824 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.359789    2571 generic.go:356] "Generic (PLEG): container finished" podID="d0e3abb4-4ab5-4b18-a497-12c812e80fb1" containerID="98bd321d32c021a66839b6759e665fd84934a1187aef0fed8a748917e8391a1f" exitCode=0
Sep 06 20:03:31.360020 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.359834    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"
Sep 06 20:03:31.360020 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.359841    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r" event={"ID":"d0e3abb4-4ab5-4b18-a497-12c812e80fb1","Type":"ContainerDied","Data":"98bd321d32c021a66839b6759e665fd84934a1187aef0fed8a748917e8391a1f"}
Sep 06 20:03:31.360020 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.359870    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r" event={"ID":"d0e3abb4-4ab5-4b18-a497-12c812e80fb1","Type":"ContainerDied","Data":"d620f477d79a709cc5c3400f9d9b897841548b95b2337a7b327f38233ef2c1e5"}
Sep 06 20:03:31.360020 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.359887    2571 scope.go:117] "RemoveContainer" containerID="98bd321d32c021a66839b6759e665fd84934a1187aef0fed8a748917e8391a1f"
Sep 06 20:03:31.369580 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.369563    2571 scope.go:117] "RemoveContainer" containerID="98bd321d32c021a66839b6759e665fd84934a1187aef0fed8a748917e8391a1f"
Sep 06 20:03:31.369837 ip-10-0-141-42 kubenswrapper[2571]: E0906 20:03:31.369819    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98bd321d32c021a66839b6759e665fd84934a1187aef0fed8a748917e8391a1f\": container with ID starting with 98bd321d32c021a66839b6759e665fd84934a1187aef0fed8a748917e8391a1f not found: ID does not exist" containerID="98bd321d32c021a66839b6759e665fd84934a1187aef0fed8a748917e8391a1f"
Sep 06 20:03:31.369885 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.369845    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98bd321d32c021a66839b6759e665fd84934a1187aef0fed8a748917e8391a1f"} err="failed to get container status \"98bd321d32c021a66839b6759e665fd84934a1187aef0fed8a748917e8391a1f\": rpc error: code = NotFound desc = could not find container \"98bd321d32c021a66839b6759e665fd84934a1187aef0fed8a748917e8391a1f\": container with ID starting with 98bd321d32c021a66839b6759e665fd84934a1187aef0fed8a748917e8391a1f not found: ID does not exist"
Sep 06 20:03:31.379374 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.379313    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"]
Sep 06 20:03:31.381241 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:31.381219    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-90c1ebd6-kserve-5f59797bc5-qbf8r"]
Sep 06 20:03:32.917714 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:32.917679    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0e3abb4-4ab5-4b18-a497-12c812e80fb1" path="/var/lib/kubelet/pods/d0e3abb4-4ab5-4b18-a497-12c812e80fb1/volumes"
Sep 06 20:03:36.264345 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.264303    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"]
Sep 06 20:03:36.264841 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.264824    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0e3abb4-4ab5-4b18-a497-12c812e80fb1" containerName="main"
Sep 06 20:03:36.264914 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.264845    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0e3abb4-4ab5-4b18-a497-12c812e80fb1" containerName="main"
Sep 06 20:03:36.265010 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.264994    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0e3abb4-4ab5-4b18-a497-12c812e80fb1" containerName="main"
Sep 06 20:03:36.269638 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.269613    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:36.273121 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.273097    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bfdfp\""
Sep 06 20:03:36.273240 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.273101    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 20:03:36.273240 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.273106    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 06 20:03:36.273399 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.273106    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 06 20:03:36.273399 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.273108    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 06 20:03:36.279482 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.279420    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"]
Sep 06 20:03:36.340176 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.340141    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-metrics-certs\") pod \"maas-api-764cc5b8c6-skc45\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") " pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:36.340348 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.340218    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhw2d\" (UniqueName: \"kubernetes.io/projected/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-kube-api-access-bhw2d\") pod \"maas-api-764cc5b8c6-skc45\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") " pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:36.340348 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.340243    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-openshift-service-ca\") pod \"maas-api-764cc5b8c6-skc45\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") " pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:36.340348 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.340317    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-maas-api-tls\") pod \"maas-api-764cc5b8c6-skc45\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") " pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:36.439608 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.439563    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:03:36.441209 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.441190    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-maas-api-tls\") pod \"maas-api-764cc5b8c6-skc45\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") " pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:36.441297 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.441241    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-metrics-certs\") pod \"maas-api-764cc5b8c6-skc45\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") " pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:36.441442 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.441418    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhw2d\" (UniqueName: \"kubernetes.io/projected/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-kube-api-access-bhw2d\") pod \"maas-api-764cc5b8c6-skc45\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") " pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:36.441490 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.441465    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-openshift-service-ca\") pod \"maas-api-764cc5b8c6-skc45\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") " pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:36.442051 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.442027    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-openshift-service-ca\") pod \"maas-api-764cc5b8c6-skc45\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") " pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:36.443681 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.443660    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-maas-api-tls\") pod \"maas-api-764cc5b8c6-skc45\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") " pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:36.443781 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.443696    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-metrics-certs\") pod \"maas-api-764cc5b8c6-skc45\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") " pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:36.448397 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.448377    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhw2d\" (UniqueName: \"kubernetes.io/projected/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-kube-api-access-bhw2d\") pod \"maas-api-764cc5b8c6-skc45\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") " pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:36.581419 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.581377    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:36.852855 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.852824    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"]
Sep 06 20:03:36.855579 ip-10-0-141-42 kubenswrapper[2571]: W0906 20:03:36.855552    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745d5d4e_8ced_4bc2_b7a7_00e2a557f127.slice/crio-33c49c3b9adf02e0ae1ff13696c07cca9b3ea9b2ef35b3061d70006166c50c0e WatchSource:0}: Error finding container 33c49c3b9adf02e0ae1ff13696c07cca9b3ea9b2ef35b3061d70006166c50c0e: Status 404 returned error can't find the container with id 33c49c3b9adf02e0ae1ff13696c07cca9b3ea9b2ef35b3061d70006166c50c0e
Sep 06 20:03:36.857315 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:36.857298    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 20:03:37.393386 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:37.393340    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45" event={"ID":"745d5d4e-8ced-4bc2-b7a7-00e2a557f127","Type":"ContainerStarted","Data":"85cd2c0782b74aa2bbd7c8341d8d6358111dfb5a4f4e914cd833710130d8294d"}
Sep 06 20:03:37.393910 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:37.393400    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45" event={"ID":"745d5d4e-8ced-4bc2-b7a7-00e2a557f127","Type":"ContainerStarted","Data":"33c49c3b9adf02e0ae1ff13696c07cca9b3ea9b2ef35b3061d70006166c50c0e"}
Sep 06 20:03:37.393910 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:37.393537    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:37.408314 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:37.408248    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45" podStartSLOduration=1.019751707 podStartE2EDuration="1.408231023s" totalImagesPullingTime="388.479316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 20:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 20:03:37.40793069 +0000 UTC m=+2180.978229039" watchObservedRunningTime="2026-09-06 20:03:37.408231023 +0000 UTC m=+2180.978529368"
Sep 06 20:03:43.404018 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:43.403981    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:44.313544 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.313505    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"]
Sep 06 20:03:44.316668 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.316648    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:44.324815 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.324760    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"]
Sep 06 20:03:44.416383 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.416328    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bcb7f0d-a82e-440d-9fea-2d6151c6e57f-maas-api-tls\") pod \"maas-api-59b97887bd-r6gbv\" (UID: \"1bcb7f0d-a82e-440d-9fea-2d6151c6e57f\") " pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:44.416383 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.416385    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9n6\" (UniqueName: \"kubernetes.io/projected/1bcb7f0d-a82e-440d-9fea-2d6151c6e57f-kube-api-access-4h9n6\") pod \"maas-api-59b97887bd-r6gbv\" (UID: \"1bcb7f0d-a82e-440d-9fea-2d6151c6e57f\") " pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:44.416869 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.416490    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcb7f0d-a82e-440d-9fea-2d6151c6e57f-metrics-certs\") pod \"maas-api-59b97887bd-r6gbv\" (UID: \"1bcb7f0d-a82e-440d-9fea-2d6151c6e57f\") " pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:44.416869 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.416615    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bcb7f0d-a82e-440d-9fea-2d6151c6e57f-openshift-service-ca\") pod \"maas-api-59b97887bd-r6gbv\" (UID: \"1bcb7f0d-a82e-440d-9fea-2d6151c6e57f\") " pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:44.518001 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.517943    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bcb7f0d-a82e-440d-9fea-2d6151c6e57f-openshift-service-ca\") pod \"maas-api-59b97887bd-r6gbv\" (UID: \"1bcb7f0d-a82e-440d-9fea-2d6151c6e57f\") " pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:44.518198 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.518020    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bcb7f0d-a82e-440d-9fea-2d6151c6e57f-maas-api-tls\") pod \"maas-api-59b97887bd-r6gbv\" (UID: \"1bcb7f0d-a82e-440d-9fea-2d6151c6e57f\") " pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:44.518198 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.518042    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9n6\" (UniqueName: \"kubernetes.io/projected/1bcb7f0d-a82e-440d-9fea-2d6151c6e57f-kube-api-access-4h9n6\") pod \"maas-api-59b97887bd-r6gbv\" (UID: \"1bcb7f0d-a82e-440d-9fea-2d6151c6e57f\") " pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:44.518198 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.518092    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcb7f0d-a82e-440d-9fea-2d6151c6e57f-metrics-certs\") pod \"maas-api-59b97887bd-r6gbv\" (UID: \"1bcb7f0d-a82e-440d-9fea-2d6151c6e57f\") " pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:44.518707 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.518684    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bcb7f0d-a82e-440d-9fea-2d6151c6e57f-openshift-service-ca\") pod \"maas-api-59b97887bd-r6gbv\" (UID: \"1bcb7f0d-a82e-440d-9fea-2d6151c6e57f\") " pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:44.520605 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.520577    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcb7f0d-a82e-440d-9fea-2d6151c6e57f-metrics-certs\") pod \"maas-api-59b97887bd-r6gbv\" (UID: \"1bcb7f0d-a82e-440d-9fea-2d6151c6e57f\") " pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:44.520712 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.520614    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bcb7f0d-a82e-440d-9fea-2d6151c6e57f-maas-api-tls\") pod \"maas-api-59b97887bd-r6gbv\" (UID: \"1bcb7f0d-a82e-440d-9fea-2d6151c6e57f\") " pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:44.525561 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.525540    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9n6\" (UniqueName: \"kubernetes.io/projected/1bcb7f0d-a82e-440d-9fea-2d6151c6e57f-kube-api-access-4h9n6\") pod \"maas-api-59b97887bd-r6gbv\" (UID: \"1bcb7f0d-a82e-440d-9fea-2d6151c6e57f\") " pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:44.629413 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.629317    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:44.834862 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.834825    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:03:44.898325 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:44.898301    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"]
Sep 06 20:03:44.900550 ip-10-0-141-42 kubenswrapper[2571]: W0906 20:03:44.900524    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bcb7f0d_a82e_440d_9fea_2d6151c6e57f.slice/crio-8534711ad7d180d64f6ef8d8c628ca88de165a0dd088632cbf7e659cce1c19a3 WatchSource:0}: Error finding container 8534711ad7d180d64f6ef8d8c628ca88de165a0dd088632cbf7e659cce1c19a3: Status 404 returned error can't find the container with id 8534711ad7d180d64f6ef8d8c628ca88de165a0dd088632cbf7e659cce1c19a3
Sep 06 20:03:45.434668 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:45.434631    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv" event={"ID":"1bcb7f0d-a82e-440d-9fea-2d6151c6e57f","Type":"ContainerStarted","Data":"30fa2f642c83eb340532740420f44c2d1ec44ec81fd6a3b01cc8d1f961317989"}
Sep 06 20:03:45.435081 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:45.434685    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv" event={"ID":"1bcb7f0d-a82e-440d-9fea-2d6151c6e57f","Type":"ContainerStarted","Data":"8534711ad7d180d64f6ef8d8c628ca88de165a0dd088632cbf7e659cce1c19a3"}
Sep 06 20:03:45.435081 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:45.434724    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:45.451971 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:45.451882    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv" podStartSLOduration=1.140807079 podStartE2EDuration="1.451864924s" totalImagesPullingTime="311.057845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 20:03:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 20:03:45.449337506 +0000 UTC m=+2189.019635853" watchObservedRunningTime="2026-09-06 20:03:45.451864924 +0000 UTC m=+2189.022163264"
Sep 06 20:03:51.445619 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.445585    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59b97887bd-r6gbv"
Sep 06 20:03:51.488418 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.488385    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"]
Sep 06 20:03:51.488634 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.488594    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45" podUID="745d5d4e-8ced-4bc2-b7a7-00e2a557f127" containerName="maas-api" containerID="cri-o://85cd2c0782b74aa2bbd7c8341d8d6358111dfb5a4f4e914cd833710130d8294d" gracePeriod=30
Sep 06 20:03:51.736644 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.736619    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:51.888480 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.888441    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-maas-api-tls\") pod \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") "
Sep 06 20:03:51.888674 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.888508    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhw2d\" (UniqueName: \"kubernetes.io/projected/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-kube-api-access-bhw2d\") pod \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") "
Sep 06 20:03:51.888674 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.888552    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-metrics-certs\") pod \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") "
Sep 06 20:03:51.888674 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.888591    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-openshift-service-ca\") pod \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\" (UID: \"745d5d4e-8ced-4bc2-b7a7-00e2a557f127\") "
Sep 06 20:03:51.889069 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.889042    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "745d5d4e-8ced-4bc2-b7a7-00e2a557f127" (UID: "745d5d4e-8ced-4bc2-b7a7-00e2a557f127"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 20:03:51.890704 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.890675    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "745d5d4e-8ced-4bc2-b7a7-00e2a557f127" (UID: "745d5d4e-8ced-4bc2-b7a7-00e2a557f127"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 20:03:51.890811 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.890730    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "745d5d4e-8ced-4bc2-b7a7-00e2a557f127" (UID: "745d5d4e-8ced-4bc2-b7a7-00e2a557f127"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 20:03:51.890811 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.890771    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-kube-api-access-bhw2d" (OuterVolumeSpecName: "kube-api-access-bhw2d") pod "745d5d4e-8ced-4bc2-b7a7-00e2a557f127" (UID: "745d5d4e-8ced-4bc2-b7a7-00e2a557f127"). InnerVolumeSpecName "kube-api-access-bhw2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 20:03:51.990212 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.990114    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-maas-api-tls\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 20:03:51.990391 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.990227    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhw2d\" (UniqueName: \"kubernetes.io/projected/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-kube-api-access-bhw2d\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 20:03:51.990391 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.990251    2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-metrics-certs\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 20:03:51.990391 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:51.990262    2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/745d5d4e-8ced-4bc2-b7a7-00e2a557f127-openshift-service-ca\") on node \"ip-10-0-141-42.ec2.internal\" DevicePath \"\""
Sep 06 20:03:52.471238 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:52.471200    2571 generic.go:356] "Generic (PLEG): container finished" podID="745d5d4e-8ced-4bc2-b7a7-00e2a557f127" containerID="85cd2c0782b74aa2bbd7c8341d8d6358111dfb5a4f4e914cd833710130d8294d" exitCode=0
Sep 06 20:03:52.471682 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:52.471244    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"
Sep 06 20:03:52.471682 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:52.471283    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45" event={"ID":"745d5d4e-8ced-4bc2-b7a7-00e2a557f127","Type":"ContainerDied","Data":"85cd2c0782b74aa2bbd7c8341d8d6358111dfb5a4f4e914cd833710130d8294d"}
Sep 06 20:03:52.471682 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:52.471335    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45" event={"ID":"745d5d4e-8ced-4bc2-b7a7-00e2a557f127","Type":"ContainerDied","Data":"33c49c3b9adf02e0ae1ff13696c07cca9b3ea9b2ef35b3061d70006166c50c0e"}
Sep 06 20:03:52.471682 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:52.471360    2571 scope.go:117] "RemoveContainer" containerID="85cd2c0782b74aa2bbd7c8341d8d6358111dfb5a4f4e914cd833710130d8294d"
Sep 06 20:03:52.480753 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:52.480735    2571 scope.go:117] "RemoveContainer" containerID="85cd2c0782b74aa2bbd7c8341d8d6358111dfb5a4f4e914cd833710130d8294d"
Sep 06 20:03:52.481065 ip-10-0-141-42 kubenswrapper[2571]: E0906 20:03:52.481045    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85cd2c0782b74aa2bbd7c8341d8d6358111dfb5a4f4e914cd833710130d8294d\": container with ID starting with 85cd2c0782b74aa2bbd7c8341d8d6358111dfb5a4f4e914cd833710130d8294d not found: ID does not exist" containerID="85cd2c0782b74aa2bbd7c8341d8d6358111dfb5a4f4e914cd833710130d8294d"
Sep 06 20:03:52.481139 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:52.481073    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85cd2c0782b74aa2bbd7c8341d8d6358111dfb5a4f4e914cd833710130d8294d"} err="failed to get container status \"85cd2c0782b74aa2bbd7c8341d8d6358111dfb5a4f4e914cd833710130d8294d\": rpc error: code = NotFound desc = could not find container \"85cd2c0782b74aa2bbd7c8341d8d6358111dfb5a4f4e914cd833710130d8294d\": container with ID starting with 85cd2c0782b74aa2bbd7c8341d8d6358111dfb5a4f4e914cd833710130d8294d not found: ID does not exist"
Sep 06 20:03:52.491901 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:52.491869    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"]
Sep 06 20:03:52.495505 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:52.495482    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-764cc5b8c6-skc45"]
Sep 06 20:03:52.918330 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:52.918297    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="745d5d4e-8ced-4bc2-b7a7-00e2a557f127" path="/var/lib/kubelet/pods/745d5d4e-8ced-4bc2-b7a7-00e2a557f127/volumes"
Sep 06 20:03:53.837255 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:03:53.837220    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:04:01.839784 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:01.839736    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:04:09.446075 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:09.446030    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-f5f2g"]
Sep 06 20:04:11.170921 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:11.170881    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6584bfcdf4-97x8k_d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5/authorino/0.log"
Sep 06 20:04:17.045364 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:17.045330    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f964dcc99-npwhb_bec8c6aa-d55b-4505-9f46-740144961193/copy-manifests/0.log"
Sep 06 20:04:17.051765 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:17.051721    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f964dcc99-npwhb_bec8c6aa-d55b-4505-9f46-740144961193/manager/0.log"
Sep 06 20:04:17.431136 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:17.431102    2571 scope.go:117] "RemoveContainer" containerID="fe4f4ae93df65b5c0d7707f25424b060f3f9612f402f74e2e46c4b680420f028"
Sep 06 20:04:17.439677 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:17.439656    2571 scope.go:117] "RemoveContainer" containerID="d5d7a2429b3a65d8b000719632245862bddd42c27915e4d91c490551d07d26a2"
Sep 06 20:04:17.491053 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:17.491027    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-59b97887bd-r6gbv_1bcb7f0d-a82e-440d-9fea-2d6151c6e57f/maas-api/0.log"
Sep 06 20:04:17.591303 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:17.591274    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29812065-glrj6_27ad3d00-06f8-4cfd-8fbd-386f026f0175/cleanup/0.log"
Sep 06 20:04:18.548051 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:18.548019    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6584bfcdf4-97x8k_d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5/authorino/0.log"
Sep 06 20:04:19.092386 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:19.092358    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-f5f2g_d41b9aee-d19d-4b66-82c3-5be20b9c9718/limitador/0.log"
Sep 06 20:04:19.871278 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:19.871249    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d55646b9-qk9vv_96e685d1-d3e3-43b8-b79b-88159bde0379/kube-auth-proxy/0.log"
Sep 06 20:04:19.992523 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:19.992492    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-95l97_c7c1898a-3846-4250-8601-fbc9bb7fcc70/istio-proxy/1.log"
Sep 06 20:04:21.091153 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:21.091119    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7m8v8_b99fa825-2680-4d0e-b9b4-abe3f01c967d/main/0.log"
Sep 06 20:04:21.309736 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:21.309703    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-llp2n_e7fd76f8-8e82-4d03-9bbc-ef42b3b29482/main/0.log"
Sep 06 20:04:30.011484 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:30.011452    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mp9fn_e5e01f7a-8ff8-4821-b662-aed4b0d11377/global-pull-secret-syncer/0.log"
Sep 06 20:04:30.166584 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:30.166547    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rkkmb_349a592c-1bd1-461a-912a-25932cf842cc/konnectivity-agent/0.log"
Sep 06 20:04:30.229064 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:30.229029    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-42.ec2.internal_46e2ca63eb83f4b4b12734788288736a/haproxy/0.log"
Sep 06 20:04:35.466270 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:35.466236    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6584bfcdf4-97x8k_d1f4450a-c494-4f8a-b0ce-7d1ded4d6bd5/authorino/0.log"
Sep 06 20:04:35.637041 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:35.636989    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-f5f2g_d41b9aee-d19d-4b66-82c3-5be20b9c9718/limitador/0.log"
Sep 06 20:04:37.302784 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:37.302750    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91955a6f-f51f-4bfd-8647-776c9c7e2a8b/alertmanager/0.log"
Sep 06 20:04:37.326931 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:37.326901    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91955a6f-f51f-4bfd-8647-776c9c7e2a8b/config-reloader/0.log"
Sep 06 20:04:37.352853 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:37.352823    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91955a6f-f51f-4bfd-8647-776c9c7e2a8b/kube-rbac-proxy-web/0.log"
Sep 06 20:04:37.387570 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:37.387539    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91955a6f-f51f-4bfd-8647-776c9c7e2a8b/kube-rbac-proxy/0.log"
Sep 06 20:04:37.406526 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:37.406491    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91955a6f-f51f-4bfd-8647-776c9c7e2a8b/kube-rbac-proxy-metric/0.log"
Sep 06 20:04:37.437152 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:37.437126    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91955a6f-f51f-4bfd-8647-776c9c7e2a8b/prom-label-proxy/0.log"
Sep 06 20:04:37.461163 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:37.461132    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_91955a6f-f51f-4bfd-8647-776c9c7e2a8b/init-config-reloader/0.log"
Sep 06 20:04:37.603776 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:37.603739    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6d889b7db9-gtpf5_74fda403-41c5-48fe-92a1-32b1c297c314/metrics-server/0.log"
Sep 06 20:04:37.800714 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:37.800683    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z46nb_2864b130-b558-40c7-a2c7-631d0cd7f342/node-exporter/0.log"
Sep 06 20:04:37.820013 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:37.819986    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z46nb_2864b130-b558-40c7-a2c7-631d0cd7f342/kube-rbac-proxy/0.log"
Sep 06 20:04:37.839262 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:37.839238    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z46nb_2864b130-b558-40c7-a2c7-631d0cd7f342/init-textfile/0.log"
Sep 06 20:04:37.964819 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:37.964787    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_65d0c889-3434-4cf5-87a4-6ae31842fd29/prometheus/0.log"
Sep 06 20:04:37.981682 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:37.981586    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_65d0c889-3434-4cf5-87a4-6ae31842fd29/config-reloader/0.log"
Sep 06 20:04:37.999596 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:37.999566    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_65d0c889-3434-4cf5-87a4-6ae31842fd29/thanos-sidecar/0.log"
Sep 06 20:04:38.024862 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:38.024832    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_65d0c889-3434-4cf5-87a4-6ae31842fd29/kube-rbac-proxy-web/0.log"
Sep 06 20:04:38.047365 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:38.047338    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_65d0c889-3434-4cf5-87a4-6ae31842fd29/kube-rbac-proxy/0.log"
Sep 06 20:04:38.063176 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:38.063147    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_65d0c889-3434-4cf5-87a4-6ae31842fd29/kube-rbac-proxy-thanos/0.log"
Sep 06 20:04:38.084164 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:38.084088    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_65d0c889-3434-4cf5-87a4-6ae31842fd29/init-config-reloader/0.log"
Sep 06 20:04:38.111096 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:38.111065    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-rw5l6_b2587a67-5897-4d57-b72b-7cfcf18b6377/prometheus-operator/0.log"
Sep 06 20:04:38.126626 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:38.126585    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-rw5l6_b2587a67-5897-4d57-b72b-7cfcf18b6377/kube-rbac-proxy/0.log"
Sep 06 20:04:38.150072 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:38.150037    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-qrrht_f8b821a1-8324-47a9-afda-fc17281c03df/prometheus-operator-admission-webhook/0.log"
Sep 06 20:04:38.177372 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:38.177338    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6df9f98f87-jgzb5_81670dcf-0538-45c8-897d-91fd12a87bea/telemeter-client/0.log"
Sep 06 20:04:38.195192 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:38.195166    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6df9f98f87-jgzb5_81670dcf-0538-45c8-897d-91fd12a87bea/reload/0.log"
Sep 06 20:04:38.212756 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:38.212712    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6df9f98f87-jgzb5_81670dcf-0538-45c8-897d-91fd12a87bea/kube-rbac-proxy/0.log"
Sep 06 20:04:40.184459 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.184428    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/1.log"
Sep 06 20:04:40.192896 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.192867    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-bngcs_87c7f616-db47-42bd-b106-8231fff3aa41/console-operator/2.log"
Sep 06 20:04:40.444451 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.444372    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"]
Sep 06 20:04:40.444749 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.444736    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745d5d4e-8ced-4bc2-b7a7-00e2a557f127" containerName="maas-api"
Sep 06 20:04:40.444791 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.444750    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="745d5d4e-8ced-4bc2-b7a7-00e2a557f127" containerName="maas-api"
Sep 06 20:04:40.444835 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.444823    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="745d5d4e-8ced-4bc2-b7a7-00e2a557f127" containerName="maas-api"
Sep 06 20:04:40.448224 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.448204    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.450757 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.450734    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4b6gr\"/\"openshift-service-ca.crt\""
Sep 06 20:04:40.451863 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.451843    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4b6gr\"/\"default-dockercfg-g8zxj\""
Sep 06 20:04:40.452025 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.451904    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4b6gr\"/\"kube-root-ca.crt\""
Sep 06 20:04:40.457377 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.457352    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"]
Sep 06 20:04:40.555608 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.555571    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spmxx\" (UniqueName: \"kubernetes.io/projected/f3f1954a-966c-4088-9e18-ac9df237a373-kube-api-access-spmxx\") pod \"perf-node-gather-daemonset-qzhpj\" (UID: \"f3f1954a-966c-4088-9e18-ac9df237a373\") " pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.555608 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.555616    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3f1954a-966c-4088-9e18-ac9df237a373-podres\") pod \"perf-node-gather-daemonset-qzhpj\" (UID: \"f3f1954a-966c-4088-9e18-ac9df237a373\") " pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.555843 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.555678    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3f1954a-966c-4088-9e18-ac9df237a373-lib-modules\") pod \"perf-node-gather-daemonset-qzhpj\" (UID: \"f3f1954a-966c-4088-9e18-ac9df237a373\") " pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.555843 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.555748    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3f1954a-966c-4088-9e18-ac9df237a373-sys\") pod \"perf-node-gather-daemonset-qzhpj\" (UID: \"f3f1954a-966c-4088-9e18-ac9df237a373\") " pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.555843 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.555808    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3f1954a-966c-4088-9e18-ac9df237a373-proc\") pod \"perf-node-gather-daemonset-qzhpj\" (UID: \"f3f1954a-966c-4088-9e18-ac9df237a373\") " pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.656811 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.656771    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3f1954a-966c-4088-9e18-ac9df237a373-lib-modules\") pod \"perf-node-gather-daemonset-qzhpj\" (UID: \"f3f1954a-966c-4088-9e18-ac9df237a373\") " pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.657027 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.656829    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3f1954a-966c-4088-9e18-ac9df237a373-sys\") pod \"perf-node-gather-daemonset-qzhpj\" (UID: \"f3f1954a-966c-4088-9e18-ac9df237a373\") " pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.657027 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.656858    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3f1954a-966c-4088-9e18-ac9df237a373-proc\") pod \"perf-node-gather-daemonset-qzhpj\" (UID: \"f3f1954a-966c-4088-9e18-ac9df237a373\") " pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.657027 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.656913    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spmxx\" (UniqueName: \"kubernetes.io/projected/f3f1954a-966c-4088-9e18-ac9df237a373-kube-api-access-spmxx\") pod \"perf-node-gather-daemonset-qzhpj\" (UID: \"f3f1954a-966c-4088-9e18-ac9df237a373\") " pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.657027 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.656942    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3f1954a-966c-4088-9e18-ac9df237a373-podres\") pod \"perf-node-gather-daemonset-qzhpj\" (UID: \"f3f1954a-966c-4088-9e18-ac9df237a373\") " pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.657027 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.656995    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3f1954a-966c-4088-9e18-ac9df237a373-proc\") pod \"perf-node-gather-daemonset-qzhpj\" (UID: \"f3f1954a-966c-4088-9e18-ac9df237a373\") " pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.657027 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.656994    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3f1954a-966c-4088-9e18-ac9df237a373-lib-modules\") pod \"perf-node-gather-daemonset-qzhpj\" (UID: \"f3f1954a-966c-4088-9e18-ac9df237a373\") " pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.657245 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.656994    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3f1954a-966c-4088-9e18-ac9df237a373-sys\") pod \"perf-node-gather-daemonset-qzhpj\" (UID: \"f3f1954a-966c-4088-9e18-ac9df237a373\") " pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.657245 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.657093    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3f1954a-966c-4088-9e18-ac9df237a373-podres\") pod \"perf-node-gather-daemonset-qzhpj\" (UID: \"f3f1954a-966c-4088-9e18-ac9df237a373\") " pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.669576 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.669551    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spmxx\" (UniqueName: \"kubernetes.io/projected/f3f1954a-966c-4088-9e18-ac9df237a373-kube-api-access-spmxx\") pod \"perf-node-gather-daemonset-qzhpj\" (UID: \"f3f1954a-966c-4088-9e18-ac9df237a373\") " pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.760344 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.760249    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:40.819131 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:40.819102    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-wvkbc_9557b720-35be-4861-8d3a-c82a3acf997e/download-server/0.log"
Sep 06 20:04:41.036230 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:41.035100    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"]
Sep 06 20:04:41.714969 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:41.714918    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj" event={"ID":"f3f1954a-966c-4088-9e18-ac9df237a373","Type":"ContainerStarted","Data":"b248ad82fd69c8c42659e9003079f5e6a5e5978e22e634dc32c0cc42c2835edd"}
Sep 06 20:04:41.715423 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:41.714992    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj" event={"ID":"f3f1954a-966c-4088-9e18-ac9df237a373","Type":"ContainerStarted","Data":"adc36c08046750598f0cfe57e97fcfdc9a9aa9c04b5cfbf4654a0b091e460f80"}
Sep 06 20:04:41.715423 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:41.715111    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:41.729135 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:41.729066    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj" podStartSLOduration=1.729053613 podStartE2EDuration="1.729053613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 20:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 20:04:41.72831433 +0000 UTC m=+2245.298612677" watchObservedRunningTime="2026-09-06 20:04:41.729053613 +0000 UTC m=+2245.299352003"
Sep 06 20:04:42.635521 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:42.635488    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nfvjj_7ef82340-6516-43c2-b523-c40361c19560/dns/0.log"
Sep 06 20:04:42.652023 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:42.651994    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nfvjj_7ef82340-6516-43c2-b523-c40361c19560/kube-rbac-proxy/0.log"
Sep 06 20:04:42.670713 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:42.670687    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fs4k9_684b35b0-78f1-4eb5-9d65-2a5b5caf6464/dns-node-resolver/0.log"
Sep 06 20:04:43.293529 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:43.293497    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z9s22_bf11e6d5-1dfb-4b4c-b01a-f5ad0c6fc91f/node-ca/0.log"
Sep 06 20:04:44.793861 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:44.793834    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d55646b9-qk9vv_96e685d1-d3e3-43b8-b79b-88159bde0379/kube-auth-proxy/0.log"
Sep 06 20:04:44.817781 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:44.817749    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-95l97_c7c1898a-3846-4250-8601-fbc9bb7fcc70/istio-proxy/0.log"
Sep 06 20:04:44.835635 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:44.835598    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-95l97_c7c1898a-3846-4250-8601-fbc9bb7fcc70/istio-proxy/1.log"
Sep 06 20:04:45.812935 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:45.812905    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sr895_f4039f06-ffbd-4400-941c-beb1104ea6f1/serve-healthcheck-canary/0.log"
Sep 06 20:04:46.415366 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:46.415339    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vj2gh_144dc9aa-3055-4442-9f9b-c9290a2d48e3/kube-rbac-proxy/0.log"
Sep 06 20:04:46.431775 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:46.431749    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vj2gh_144dc9aa-3055-4442-9f9b-c9290a2d48e3/exporter/0.log"
Sep 06 20:04:46.449077 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:46.449049    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vj2gh_144dc9aa-3055-4442-9f9b-c9290a2d48e3/extractor/0.log"
Sep 06 20:04:47.730201 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:47.730172    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4b6gr/perf-node-gather-daemonset-qzhpj"
Sep 06 20:04:49.233433 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:49.233397    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f964dcc99-npwhb_bec8c6aa-d55b-4505-9f46-740144961193/manager/0.log"
Sep 06 20:04:49.251161 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:49.251135    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f964dcc99-npwhb_bec8c6aa-d55b-4505-9f46-740144961193/copy-manifests/0.log"
Sep 06 20:04:50.709884 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:50.709795    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7769675c48-rn4w6_03a1c734-f7c7-45d2-9868-eb943d7f28c9/manager/0.log"
Sep 06 20:04:51.942321 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:51.942290    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d55646b9-qk9vv_96e685d1-d3e3-43b8-b79b-88159bde0379/kube-auth-proxy/0.log"
Sep 06 20:04:51.962640 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:51.962597    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-95l97_c7c1898a-3846-4250-8601-fbc9bb7fcc70/istio-proxy/0.log"
Sep 06 20:04:51.980794 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:51.980755    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-95l97_c7c1898a-3846-4250-8601-fbc9bb7fcc70/istio-proxy/1.log"
Sep 06 20:04:54.159107 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:54.159070    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d55646b9-qk9vv_96e685d1-d3e3-43b8-b79b-88159bde0379/kube-auth-proxy/0.log"
Sep 06 20:04:54.180072 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:54.180032    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-95l97_c7c1898a-3846-4250-8601-fbc9bb7fcc70/istio-proxy/0.log"
Sep 06 20:04:54.197706 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:54.197678    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-95l97_c7c1898a-3846-4250-8601-fbc9bb7fcc70/istio-proxy/1.log"
Sep 06 20:04:57.108954 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:57.108927    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xp729_e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94/kube-multus-additional-cni-plugins/0.log"
Sep 06 20:04:57.126014 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:57.125976    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xp729_e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94/egress-router-binary-copy/0.log"
Sep 06 20:04:57.142398 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:57.142373    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xp729_e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94/cni-plugins/0.log"
Sep 06 20:04:57.158866 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:57.158835    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xp729_e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94/bond-cni-plugin/0.log"
Sep 06 20:04:57.174813 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:57.174789    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xp729_e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94/routeoverride-cni/0.log"
Sep 06 20:04:57.190953 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:57.190931    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xp729_e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94/whereabouts-cni-bincopy/0.log"
Sep 06 20:04:57.208021 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:57.207998    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xp729_e8e8b6e7-c76c-4775-8a5c-4594f0a1ca94/whereabouts-cni/0.log"
Sep 06 20:04:57.313502 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:57.313471    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qrvg8_00e9861f-8b5d-4ed4-b1be-777aa3ae2d0a/kube-multus/0.log"
Sep 06 20:04:57.475472 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:57.475391    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pghst_abf4350f-c25e-43ae-9271-5987625590f7/network-metrics-daemon/0.log"
Sep 06 20:04:57.488269 ip-10-0-141-42 kubenswrapper[2571]: I0906 20:04:57.488242    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pghst_abf4350f-c25e-43ae-9271-5987625590f7/kube-rbac-proxy/0.log"