Sep 18 18:38:59.164483 ip-10-0-130-78 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 18:39:02.414254 ip-10-0-130-78 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 18:39:02.414254 ip-10-0-130-78 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 18:39:02.414254 ip-10-0-130-78 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 18:39:02.414254 ip-10-0-130-78 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 18:39:02.414254 ip-10-0-130-78 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 18:39:02.414927 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.414356    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 18:39:02.418243 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418228    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:39:02.418243 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418243    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418247    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418251    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418254    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418257    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418260    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418263    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418265    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418268    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418271    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418273    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418276    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418279    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418281    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418284    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418286    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418289    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418292    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418294    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418297    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:39:02.418308 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418299    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418303    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418305    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418308    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418311    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418314    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418316    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418319    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418322    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418325    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418327    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418330    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418332    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418335    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418338    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418340    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418343    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418347    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418351    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418354    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:39:02.418828 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418357    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418359    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418362    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418365    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418367    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418369    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418372    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418375    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418378    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418380    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418383    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418385    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418388    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418391    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418394    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418397    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418400    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418402    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418405    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418408    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:39:02.419341 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418411    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418413    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418416    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418419    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418423    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418428    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418431    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418434    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418437    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418440    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418442    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418445    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418448    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418451    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418453    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418457    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418461    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418464    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418468    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:39:02.419822 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418471    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418474    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418477    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418480    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418483    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418487    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418490    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418494    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418497    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418499    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418502    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418505    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418507    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418510    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418513    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418515    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418517    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418520    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.418523    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:39:02.420289 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419207    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419224    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419227    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419230    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419232    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419235    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419238    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419240    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419243    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419246    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419249    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419251    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419254    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419257    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419260    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419263    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419267    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419271    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419275    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:39:02.420757 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419278    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419282    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419285    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419288    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419291    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419294    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419296    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419299    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419302    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419305    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419308    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419311    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419313    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419316    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419319    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419321    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419324    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419326    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419329    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:39:02.421279 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419332    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419334    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419337    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419340    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419343    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419345    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419348    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419351    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419354    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419357    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419360    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419363    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419366    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419369    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419372    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419375    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419378    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419381    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419384    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419387    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:39:02.421745 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419390    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419392    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419395    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419398    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419401    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419404    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419407    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419410    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419412    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419415    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419417    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419421    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419425    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419428    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419431    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419434    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419437    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419440    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419443    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419446    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:39:02.422236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419449    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419452    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419455    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419457    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419460    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419463    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419465    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419468    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419472    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419474    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419477    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419480    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419482    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419485    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419487    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419490    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419493    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419495    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419498    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419500    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:39:02.422719 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419503    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419857    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419862    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419865    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419868    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419872    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419875    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419877    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419880    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419883    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419885    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419889    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419892    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419895    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419898    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419900    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419903    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419906    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419909    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419911    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:39:02.423229 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419914    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419916    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419919    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419921    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419924    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419926    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419929    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419933    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419937    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419939    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419943    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419946    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419948    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419951    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419954    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419957    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419962    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419965    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419968    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419970    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:39:02.423713 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419973    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419976    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419978    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419981    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419984    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419986    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419989    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419992    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419994    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.419997    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420000    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420002    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420005    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420007    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420010    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420013    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420017    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420020    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420023    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:39:02.424204 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420026    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420029    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420032    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420035    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420038    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420041    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420044    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420046    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420049    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420053    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420056    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420058    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420061    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420063    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420066    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420070    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420073    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420076    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420079    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420082    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:39:02.424655 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420086    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420088    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420091    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420094    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420096    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420099    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420102    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420105    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420107    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420110    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420112    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420115    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420117    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420120    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420122    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420125    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420127    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420130    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420133    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420136    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:39:02.425169 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420139    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420238    2558 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420282    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420290    2558 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420294    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420299    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420302    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420306    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420311    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420314    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420317    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420321    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420324    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420328    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420331    2558 flags.go:64] FLAG: --cgroup-root=""
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420334    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420337    2558 flags.go:64] FLAG: --client-ca-file=""
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420340    2558 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420343    2558 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420346    2558 flags.go:64] FLAG: --cluster-domain=""
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420349    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420353    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420356    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420359    2558 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 18:39:02.425659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420363    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420366    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420369    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420372    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420375    2558 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420378    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420381    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420385    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420388    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420394    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420398    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420401    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420403    2558 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420407    2558 flags.go:64] FLAG: --enable-server="true"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420410    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420414    2558 flags.go:64] FLAG: --event-burst="100"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420417    2558 flags.go:64] FLAG: --event-qps="50"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420420    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420423    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420426    2558 flags.go:64] FLAG: --eviction-hard=""
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420430    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420433    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420439    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420443    2558 flags.go:64] FLAG: --eviction-soft=""
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420446    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 18:39:02.426248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420448    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420451    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420454    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420457    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420460    2558 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420463    2558 flags.go:64] FLAG: --feature-gates=""
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420472    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420475    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420478    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420481    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420484    2558 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420488    2558 flags.go:64] FLAG: --help="false"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420491    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420494    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420498    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420501    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420506    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420509    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420512    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420515    2558 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420518    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420521    2558 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420524    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420527    2558 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 18:39:02.427070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420530    2558 flags.go:64] FLAG: --kube-reserved=""
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420533    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420536    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420539    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420542    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420545    2558 flags.go:64] FLAG: --lock-file=""
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420551    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420554    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420557    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420562    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420565    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420568    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420571    2558 flags.go:64] FLAG: --logging-format="text"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420574    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420577    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420580    2558 flags.go:64] FLAG: --manifest-url=""
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420582    2558 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420586    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420589    2558 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420593    2558 flags.go:64] FLAG: --max-pods="110"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420597    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420600    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420607    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420611    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420616    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 18:39:02.428119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420619    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420622    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420630    2558 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420633    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420636    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420639    2558 flags.go:64] FLAG: --pod-cidr=""
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420641    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420647    2558 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420650    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420653    2558 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420655    2558 flags.go:64] FLAG: --port="10250"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420659    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420662    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-033edc514c004af57"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420665    2558 flags.go:64] FLAG: --qos-reserved=""
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420669    2558 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420672    2558 flags.go:64] FLAG: --register-node="true"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420675    2558 flags.go:64] FLAG: --register-with-taints=""
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420678    2558 flags.go:64] FLAG: --registry-burst="10"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420681    2558 flags.go:64] FLAG: --registry-qps="5"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420684    2558 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420687    2558 flags.go:64] FLAG: --reserved-memory=""
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420691    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420694    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420697    2558 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420700    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 18:39:02.429171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420703    2558 flags.go:64] FLAG: --runonce="false"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420705    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420708    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420711    2558 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420716    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420719    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420722    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420726    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420730    2558 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420733    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420736    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420738    2558 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420741    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420744    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420748    2558 flags.go:64] FLAG: --system-cgroups=""
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420750    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420756    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420758    2558 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420762    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420766    2558 flags.go:64] FLAG: --tls-min-version=""
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420769    2558 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420772    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420775    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420778    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420781    2558 flags.go:64] FLAG: --v="2"
Sep 18 18:39:02.430294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420785    2558 flags.go:64] FLAG: --version="false"
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420789    2558 flags.go:64] FLAG: --vmodule=""
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420792    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.420796    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420905    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420909    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420913    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420916    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420919    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420921    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420924    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420929    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420932    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420934    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420938    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420941    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420943    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420946    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420949    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420952    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:39:02.431042 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420954    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420956    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420959    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420962    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420964    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420967    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420970    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420972    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420975    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420977    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420980    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420983    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420985    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420988    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420990    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420993    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420995    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.420998    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421000    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421003    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:39:02.431881 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421006    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421008    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421011    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421015    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421018    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421020    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421023    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421027    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421031    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421034    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421037    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421040    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421042    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421045    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421048    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421050    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421053    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421056    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421058    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:39:02.432510 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421061    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421064    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421066    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421069    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421072    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421074    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421077    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421079    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421083    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421086    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421089    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421092    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421094    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421097    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421101    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421104    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421107    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421110    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421113    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421116    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:39:02.433016 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421119    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421121    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421124    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421126    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421129    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421131    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421134    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421136    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421139    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421142    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421144    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421148    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421151    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421154    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421156    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421159    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421161    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421165    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421167    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:39:02.433683 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421170    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:39:02.434219 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421172    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:39:02.434219 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421175    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:39:02.434219 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421177    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:39:02.434219 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.421180    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:39:02.434219 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.421199    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 18:39:02.434219 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.428923    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 18:39:02.434219 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.428943    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 18:39:02.434219 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.428970    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 18:39:02.434219 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.428984    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 18:39:02.434219 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429066    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:39:02.434219 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429073    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:39:02.434219 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429079    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:39:02.434219 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429083    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:39:02.434219 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429088    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:39:02.434219 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429092    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429098    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429103    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429110    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429116    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429121    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429128    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429133    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429138    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429144    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429149    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429154    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429159    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429163    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429167    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429171    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429175    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429179    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429199    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:39:02.434726 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429203    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429207    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429211    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429215    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429220    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429226    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429230    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429234    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429238    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429243    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429247    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429252    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429256    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429260    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429264    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429268    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429272    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429277    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429281    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429285    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:39:02.435203 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429290    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429294    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429299    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429303    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429307    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429311    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429315    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429319    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429323    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429328    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429332    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429336    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429340    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429345    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429349    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429353    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429357    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429361    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429366    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429370    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:39:02.435718 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429375    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429379    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429383    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429387    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429391    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429395    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429399    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429403    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429408    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429412    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429416    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429420    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429425    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429428    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429432    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429437    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429441    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429445    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429449    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429454    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:39:02.436326 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429458    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:39:02.436830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429462    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:39:02.436830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429466    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:39:02.436830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429470    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:39:02.436830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429474    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:39:02.436830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429478    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:39:02.436830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429482    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:39:02.436830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429486    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:39:02.436830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429491    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:39:02.436830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429495    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:39:02.436830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429499    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:39:02.436830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429505    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:39:02.436830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429509    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:39:02.436830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429513    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:39:02.436830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429519    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:39:02.437210 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.429528    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 18:39:02.437210 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429742    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:39:02.437210 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429750    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:39:02.437210 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429756    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:39:02.437210 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429760    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:39:02.437210 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429765    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:39:02.437210 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429769    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:39:02.437210 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429773    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:39:02.437210 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429777    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:39:02.437210 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429781    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:39:02.437210 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429786    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:39:02.437210 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429790    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:39:02.437210 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429794    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:39:02.437210 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429799    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:39:02.437210 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429803    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429807    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429811    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429815    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429820    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429824    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429829    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429833    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429837    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429842    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429846    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429850    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429855    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429859    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429863    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429869    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429873    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429880    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429900    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429906    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:39:02.437830 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429911    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429916    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429920    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429925    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429930    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429934    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429939    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429943    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429949    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429954    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429958    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429963    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429967    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429971    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429975    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429979    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429983    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429987    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429991    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:39:02.438681 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429995    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.429999    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430004    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430008    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430012    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430016    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430021    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430024    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430028    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430034    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430038    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430042    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430046    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430051    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430055    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430062    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430068    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430073    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430077    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430082    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:39:02.439484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430086    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430091    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430096    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430101    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430105    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430110    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430114    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430118    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430123    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430127    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430131    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430135    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430139    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430143    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430147    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430151    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430155    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430159    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430163    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430168    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:39:02.440246 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430172    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:39:02.440739 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430176    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:39:02.440739 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430181    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:39:02.440739 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430203    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:39:02.440739 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430207    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:39:02.440739 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430211    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:39:02.440739 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:02.430215    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:39:02.440739 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.430223    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 18:39:02.440739 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.430375    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 18:39:02.440739 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.434360    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 18:39:02.440739 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.435199    2558 server.go:1019] "Starting client certificate rotation"
Sep 18 18:39:02.440739 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.435295    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 18:39:02.440739 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.435325    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 18:39:02.440739 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.437795    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 18:39:02.441077 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.439637    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 18:39:02.446297 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.446276    2558 log.go:25] "Validated CRI v1 runtime API"
Sep 18 18:39:02.447853 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.447832    2558 log.go:25] "Validated CRI v1 image API"
Sep 18 18:39:02.448561 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.448547    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 18:39:02.450700 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.450679    2558 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c1c449bc-41ed-4098-b326-1065fe9ddab2:/dev/nvme0n1p3 d592530d-e098-40fa-aff9-abea25dbb25c:/dev/nvme0n1p4]
Sep 18 18:39:02.450823 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.450697    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 18:39:02.454851 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.454737    2558 manager.go:216] Machine: {Timestamp:2026-09-18 18:39:02.454263262 +0000 UTC m=+1.890842095 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098299 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec247564e6fce1df9e9e34f4720520a6 SystemUUID:ec247564-e6fc-e1df-9e9e-34f4720520a6 BootID:a0fe6c6a-d862-4647-83c0-72398cf434c2 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:41:d6:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:41:d6:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:ea:e6:c1:79:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 18:39:02.454851 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.454840    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 18:39:02.454969 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.454909    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 18:39:02.456369 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.456340    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 18:39:02.456521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.456370    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-78.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 18:39:02.457672 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.457661    2558 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 18:39:02.457717 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.457674    2558 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 18:39:02.457717 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.457682    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 18:39:02.457717 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.457692    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 18:39:02.457800 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.457758    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 18:39:02.462194 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.462173    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 18:39:02.462310 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.462300    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 18:39:02.462347 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.462337    2558 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 18:39:02.462376 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.462347    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 18:39:02.462376 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.462363    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 18:39:02.462376 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.462372    2558 kubelet.go:398] "Adding apiserver pod source"
Sep 18 18:39:02.462463 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.462380    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 18:39:02.462698 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.462682    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 18:39:02.462752 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.462702    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 18:39:02.464599 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.464579    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 18:39:02.466398 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.466385    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 18:39:02.466443 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.466415    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 18:39:02.467614 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.467601    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 18:39:02.467651 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.467625    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 18:39:02.467651 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.467631    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 18:39:02.467651 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.467637    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 18:39:02.467651 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.467646    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 18:39:02.467651 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.467651    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 18:39:02.467780 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.467665    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 18:39:02.467780 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.467670    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 18:39:02.467780 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.467676    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 18:39:02.467780 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.467682    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 18:39:02.467780 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.467695    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 18:39:02.467780 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.467700    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 18:39:02.467780 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.467723    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 18:39:02.467780 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.467728    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 18:39:02.468625 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.468609    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 18:39:02.471760 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.471745    2558 server.go:1268] "Started kubelet"
Sep 18 18:39:02.471846 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.471821    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 18:39:02.471897 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.471834    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 18:39:02.471954 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.471919    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 18:39:02.472657 ip-10-0-130-78 systemd[1]: Started Kubernetes Kubelet.
Sep 18 18:39:02.473264 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.473124    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 18:39:02.474251 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.474235    2558 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 18:39:02.499268 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.499236    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 18:39:02.499998 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.499969    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-78.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 18:39:02.500080 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.500015    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 18:39:02.500169 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.500149    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-78.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 18:39:02.503331 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.503315    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 18:39:02.503411 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.503337    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 18:39:02.503459 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.503423    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-78.ec2.internal\" not found"
Sep 18 18:39:02.503937 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.503867    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 18:39:02.503937 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.503896    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 18:39:02.503937 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.503867    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 18:39:02.503937 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.502992    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-78.ec2.internal.18d67e465561e3d8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-78.ec2.internal,UID:ip-10-0-130-78.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-78.ec2.internal,},FirstTimestamp:2026-09-18 18:39:02.471640024 +0000 UTC m=+1.908218857,LastTimestamp:2026-09-18 18:39:02.471640024 +0000 UTC m=+1.908218857,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-78.ec2.internal,}"
Sep 18 18:39:02.504143 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.504034    2558 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 18:39:02.504143 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.504051    2558 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 18:39:02.504330 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.504309    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 18:39:02.504330 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.504330    2558 factory.go:55] Registering systemd factory
Sep 18 18:39:02.504484 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.504339    2558 factory.go:223] Registration of the systemd container factory successfully
Sep 18 18:39:02.504560 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.504547    2558 factory.go:153] Registering CRI-O factory
Sep 18 18:39:02.504629 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.504565    2558 factory.go:223] Registration of the crio container factory successfully
Sep 18 18:39:02.504629 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.504582    2558 factory.go:103] Registering Raw factory
Sep 18 18:39:02.504629 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.504606    2558 manager.go:1196] Started watching for new ooms in manager
Sep 18 18:39:02.505044 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.505018    2558 manager.go:349] Starting recovery of all containers
Sep 18 18:39:02.512399 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.512373    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 18:39:02.512399 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.512385    2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-78.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 18:39:02.515377 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.515360    2558 manager.go:354] Recovery completed
Sep 18 18:39:02.518537 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.518505    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jgpxk"
Sep 18 18:39:02.519418 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.519407    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:39:02.520997 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.520982    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:39:02.521049 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.521009    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:39:02.521049 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.521020    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:39:02.521537 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.521519    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 18:39:02.521537 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.521534    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 18:39:02.521693 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.521543    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:39:02.521693 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.521552    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 18:39:02.522774 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.522754    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:39:02.522846 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.522786    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:39:02.522846 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.522799    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:39:02.523807 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.523795    2558 policy_none.go:49] "None policy: Start"
Sep 18 18:39:02.523846 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.523811    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 18:39:02.523846 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.523821    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 18:39:02.524521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.524508    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jgpxk"
Sep 18 18:39:02.525889 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.525876    2558 policy_none.go:47] "Start"
Sep 18 18:39:02.530522 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.530440    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-78.ec2.internal.18d67e4658530508  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-78.ec2.internal,UID:ip-10-0-130-78.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-78.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-78.ec2.internal,},FirstTimestamp:2026-09-18 18:39:02.520997128 +0000 UTC m=+1.957575961,LastTimestamp:2026-09-18 18:39:02.520997128 +0000 UTC m=+1.957575961,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-78.ec2.internal,}"
Sep 18 18:39:02.574843 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.567258    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 18:39:02.574843 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.568463    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 18:39:02.574843 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.568511    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 18:39:02.574843 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.568536    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 18:39:02.574843 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.568580    2558 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 18 18:39:02.574843 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.570493    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:39:02.578768 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.578756    2558 manager.go:337] "Starting Device Plugin manager"
Sep 18 18:39:02.578816 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.578789    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 18:39:02.578816 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.578799    2558 server.go:87] "Starting device plugin registration server"
Sep 18 18:39:02.579073 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.579062    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 18:39:02.579110 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.579089    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 18:39:02.579210 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.579174    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 18:39:02.579357 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.579270    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 18:39:02.579357 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.579279    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 18:39:02.579805 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.579787    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 18:39:02.579867 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.579842    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-78.ec2.internal\" not found"
Sep 18 18:39:02.669380 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.669164    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-78.ec2.internal"]
Sep 18 18:39:02.669505 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.669390    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:39:02.671040 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.671026    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:39:02.671115 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.671053    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:39:02.671115 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.671064    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:39:02.673476 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.673463    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:39:02.673610 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.673594    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.673671 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.673643    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:39:02.674227 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.674206    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:39:02.674227 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.674225    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:39:02.674348 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.674232    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:39:02.674348 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.674246    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:39:02.674348 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.674257    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:39:02.674348 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.674247    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:39:02.676599 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.676585    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.676665 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.676610    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:39:02.677290 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.677274    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:39:02.677380 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.677303    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:39:02.677380 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.677317    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:39:02.679295 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.679281    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:39:02.680024 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.680009    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:39:02.680111 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.680038    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:39:02.680111 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.680053    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:39:02.680111 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.680079    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.687781 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.687767    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.687871 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.687789    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-78.ec2.internal\": node \"ip-10-0-130-78.ec2.internal\" not found"
Sep 18 18:39:02.699796 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.699775    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-78.ec2.internal\" not found" node="ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.704075 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.704060    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-78.ec2.internal\" not found" node="ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.705130 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.705116    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9265269b110ff616d074999444471d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal\" (UID: \"b9265269b110ff616d074999444471d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.705197 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.705138    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9265269b110ff616d074999444471d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal\" (UID: \"b9265269b110ff616d074999444471d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.705197 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.705155    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8aa0497c56ca99c73b6260acf272ac5b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-78.ec2.internal\" (UID: \"8aa0497c56ca99c73b6260acf272ac5b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.712095 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.712079    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-78.ec2.internal\" not found"
Sep 18 18:39:02.805538 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.805511    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9265269b110ff616d074999444471d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal\" (UID: \"b9265269b110ff616d074999444471d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.805640 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.805562    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9265269b110ff616d074999444471d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal\" (UID: \"b9265269b110ff616d074999444471d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.805640 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.805599    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8aa0497c56ca99c73b6260acf272ac5b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-78.ec2.internal\" (UID: \"8aa0497c56ca99c73b6260acf272ac5b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.805640 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.805611    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9265269b110ff616d074999444471d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal\" (UID: \"b9265269b110ff616d074999444471d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.805732 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.805642    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8aa0497c56ca99c73b6260acf272ac5b-config\") pod \"kube-apiserver-proxy-ip-10-0-130-78.ec2.internal\" (UID: \"8aa0497c56ca99c73b6260acf272ac5b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.805732 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:02.805660    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9265269b110ff616d074999444471d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal\" (UID: \"b9265269b110ff616d074999444471d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal"
Sep 18 18:39:02.812632 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.812615    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-78.ec2.internal\" not found"
Sep 18 18:39:02.913143 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:02.913122    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-78.ec2.internal\" not found"
Sep 18 18:39:03.003683 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.003625    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal"
Sep 18 18:39:03.007529 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.007504    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-78.ec2.internal"
Sep 18 18:39:03.013321 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:03.013303    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-78.ec2.internal\" not found"
Sep 18 18:39:03.113803 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:03.113772    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-78.ec2.internal\" not found"
Sep 18 18:39:03.214302 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:03.214272    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-78.ec2.internal\" not found"
Sep 18 18:39:03.314854 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:03.314776    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-78.ec2.internal\" not found"
Sep 18 18:39:03.415269 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:03.415224    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-78.ec2.internal\" not found"
Sep 18 18:39:03.435412 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.435384    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 18:39:03.435584 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.435563    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 18:39:03.515808 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:03.515781    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-78.ec2.internal\" not found"
Sep 18 18:39:03.518529 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.518510    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:39:03.529959 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.529933    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 18:34:02 +0000 UTC" deadline="2028-04-17 19:01:32.376973619 +0000 UTC"
Sep 18 18:39:03.530047 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.529959    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13848h22m28.847017667s"
Sep 18 18:39:03.534078 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.534064    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:39:03.534987 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.534969    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:39:03.535070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.534998    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:39:03.535070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.535007    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:39:03.540750 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.540735    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:39:03.616406 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:03.616386    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-78.ec2.internal\" not found"
Sep 18 18:39:03.716920 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:03.716898    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-78.ec2.internal\" not found"
Sep 18 18:39:03.775125 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.775104    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:39:03.804257 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.804233    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal"
Sep 18 18:39:03.817479 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.817456    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 18:39:03.818432 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.818413    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-78.ec2.internal"
Sep 18 18:39:03.827105 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.827088    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 18:39:03.897929 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:03.897873    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9265269b110ff616d074999444471d7.slice/crio-b4f19ed7f71841ab8b411a32eb134df85ce7c0cb74c142d780427f6384cd0248 WatchSource:0}: Error finding container b4f19ed7f71841ab8b411a32eb134df85ce7c0cb74c142d780427f6384cd0248: Status 404 returned error can't find the container with id b4f19ed7f71841ab8b411a32eb134df85ce7c0cb74c142d780427f6384cd0248
Sep 18 18:39:03.902586 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:03.902570    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:39:03.930278 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:03.930254    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa0497c56ca99c73b6260acf272ac5b.slice/crio-69170c558b3bd4482ddd1c2ec2026018211ca988a1dc3e4fa5486214a7351c99 WatchSource:0}: Error finding container 69170c558b3bd4482ddd1c2ec2026018211ca988a1dc3e4fa5486214a7351c99: Status 404 returned error can't find the container with id 69170c558b3bd4482ddd1c2ec2026018211ca988a1dc3e4fa5486214a7351c99
Sep 18 18:39:04.463280 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.463251    2558 apiserver.go:52] "Watching apiserver"
Sep 18 18:39:04.471372 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.471345    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 18:39:04.472482 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.472312    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-4ww5h","openshift-multus/multus-additional-cni-plugins-thdgp","openshift-network-operator/iptables-alerter-l769f","openshift-ovn-kubernetes/ovnkube-node-l6vdn","kube-system/konnectivity-agent-xmrrh","openshift-cluster-node-tuning-operator/tuned-7j542","openshift-dns/node-resolver-xbz24","openshift-image-registry/node-ca-zmgsq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal","openshift-multus/network-metrics-daemon-dtwsf","openshift-network-diagnostics/network-check-target-xrssq","kube-system/global-pull-secret-syncer-w9bz8","kube-system/kube-apiserver-proxy-ip-10-0-130-78.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"]
Sep 18 18:39:04.475237 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.475220    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:04.475314 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:04.475289    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:04.477171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.477159    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:04.477286 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:04.477210    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:04.479140 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.479124    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l769f"
Sep 18 18:39:04.481063 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.481045    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 18:39:04.481195 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.481162    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-frb9p\""
Sep 18 18:39:04.481272 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.481254    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.481366 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.481350    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 18:39:04.481456 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.481353    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 18:39:04.483206 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.483178    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 18:39:04.483329 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.483308    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 18:39:04.483329 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.483316    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 18:39:04.483457 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.483392    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vzw98\""
Sep 18 18:39:04.483557 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.483545    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xmrrh"
Sep 18 18:39:04.485517 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.485499    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-trd8p\""
Sep 18 18:39:04.485517 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.485512    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 18:39:04.485635 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.485500    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 18:39:04.485736 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.485722    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.487613 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.487598    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 18:39:04.487748 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.487733    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 18:39:04.487855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.487842    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wbt22\""
Sep 18 18:39:04.488011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.487996    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zmgsq"
Sep 18 18:39:04.490227 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.490203    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 18:39:04.490227 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.490221    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-72zz2\""
Sep 18 18:39:04.490383 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.490247    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 18:39:04.490383 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.490223    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 18:39:04.490508 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.490492    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xbz24"
Sep 18 18:39:04.492456 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.492437    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 18:39:04.492843 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.492711    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.492843 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.492775    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 18:39:04.492843 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.492785    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w992t\""
Sep 18 18:39:04.494878 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.494852    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 18:39:04.494955 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.494883    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 18:39:04.494955 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.494899    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 18:39:04.494955 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.494932    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.495266 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.495249    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9qw7x\""
Sep 18 18:39:04.495266 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.495261    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 18:39:04.495385 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.495315    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 18:39:04.496988 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.496963    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 18:39:04.497118 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.497105    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mb86q\""
Sep 18 18:39:04.497237 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.497225    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:04.497309 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:04.497284    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:04.499439 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.499415    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.501575 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.501557    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 18:39:04.501677 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.501599    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 18:39:04.501677 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.501603    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 18:39:04.502511 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.502496    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kc2ts\""
Sep 18 18:39:04.502575 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.502561    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 18:39:04.502768 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.502755    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 18:39:04.502839 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.502759    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 18:39:04.503611 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.503597    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 18:39:04.504910 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.504896    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 18:39:04.512859 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.512842    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 18:39:04.514370 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514355    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4hl8\" (UniqueName: \"kubernetes.io/projected/38e19c6c-afca-4121-a242-f497fddd6827-kube-api-access-v4hl8\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.514432 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514378    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-registration-dir\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.514432 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514393    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60babd6b-456a-43cb-81c8-0967234b4c7a-system-cni-dir\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.514432 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514415    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-sysctl-conf\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.514547 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514435    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-cnibin\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.514547 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514486    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/38e19c6c-afca-4121-a242-f497fddd6827-multus-daemon-config\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.514547 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514523    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-device-dir\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.514701 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514562    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-var-lib-openvswitch\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.514701 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514590    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-system-cni-dir\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.514701 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514631    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znrh7\" (UniqueName: \"kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7\") pod \"network-check-target-xrssq\" (UID: \"d2997254-b30e-438d-b105-40088a498ee5\") " pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:04.514701 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514648    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-kubernetes\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.514701 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514663    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-run-multus-certs\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.514701 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514676    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/077f92f7-ee39-40c4-9873-43b00c9141a9-agent-certs\") pod \"konnectivity-agent-xmrrh\" (UID: \"077f92f7-ee39-40c4-9873-43b00c9141a9\") " pod="kube-system/konnectivity-agent-xmrrh"
Sep 18 18:39:04.514701 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514690    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.514991 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514711    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-ovn-node-metrics-cert\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.514991 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514725    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-kubelet\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.514991 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514737    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-ovnkube-config\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.514991 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514749    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-env-overrides\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.514991 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514765    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-run-netns\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.514991 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514800    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk75h\" (UniqueName: \"kubernetes.io/projected/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-kube-api-access-qk75h\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.514991 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514834    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-etc-openvswitch\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.514991 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514862    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-ovnkube-script-lib\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.514991 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514906    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-sys\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.514991 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514930    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aad288b6-deba-464d-917f-dcc47d794865-serviceca\") pod \"node-ca-zmgsq\" (UID: \"aad288b6-deba-464d-917f-dcc47d794865\") " pod="openshift-image-registry/node-ca-zmgsq"
Sep 18 18:39:04.514991 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514972    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-run-k8s-cni-cncf-io\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.514991 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.514994    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-hostroot\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.515392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515018    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-multus-conf-dir\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.515392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515052    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-etc-kubernetes\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.515392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515086    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.515392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515121    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-socket-dir\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.515392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515165    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-multus-cni-dir\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.515392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515207    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3295807d-bb8e-4948-9a6a-e438b0f4e048-tmp\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.515392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515247    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-multus-socket-dir-parent\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.515392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515283    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-var-lib-kubelet\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.515392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515313    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-slash\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.515392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515329    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.515392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515345    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s892\" (UniqueName: \"kubernetes.io/projected/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-kube-api-access-7s892\") pod \"network-metrics-daemon-dtwsf\" (UID: \"2d9a33e8-d5ff-482e-b822-5aa13b177a6f\") " pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:04.515392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515369    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-tuned\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.515392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515384    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21bb6420-c1bc-4591-b804-4a69aa5092d6-iptables-alerter-script\") pod \"iptables-alerter-l769f\" (UID: \"21bb6420-c1bc-4591-b804-4a69aa5092d6\") " pod="openshift-network-operator/iptables-alerter-l769f"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515401    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-systemd-units\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515420    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515433    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60babd6b-456a-43cb-81c8-0967234b4c7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515446    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-run-ovn\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515464    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-systemd\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515486    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz2wg\" (UniqueName: \"kubernetes.io/projected/3295807d-bb8e-4948-9a6a-e438b0f4e048-kube-api-access-kz2wg\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515518    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njxzv\" (UniqueName: \"kubernetes.io/projected/60babd6b-456a-43cb-81c8-0967234b4c7a-kube-api-access-njxzv\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515554    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9ac6087e-296b-4917-bee6-8540193b8719-dbus\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515580    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-run-netns\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515602    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-sysconfig\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515622    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-host\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515644    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21bb6420-c1bc-4591-b804-4a69aa5092d6-host-slash\") pod \"iptables-alerter-l769f\" (UID: \"21bb6420-c1bc-4591-b804-4a69aa5092d6\") " pod="openshift-network-operator/iptables-alerter-l769f"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515669    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lwgk\" (UniqueName: \"kubernetes.io/projected/21bb6420-c1bc-4591-b804-4a69aa5092d6-kube-api-access-4lwgk\") pod \"iptables-alerter-l769f\" (UID: \"21bb6420-c1bc-4591-b804-4a69aa5092d6\") " pod="openshift-network-operator/iptables-alerter-l769f"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515694    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60babd6b-456a-43cb-81c8-0967234b4c7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515718    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-run-systemd\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.515777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515740    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-cni-bin\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515763    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-cni-netd\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515785    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-run\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515838    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-lib-modules\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515863    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa59b9bc-694f-4b16-99bf-5d31a85bca74-hosts-file\") pod \"node-resolver-xbz24\" (UID: \"aa59b9bc-694f-4b16-99bf-5d31a85bca74\") " pod="openshift-dns/node-resolver-xbz24"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515886    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8bxh\" (UniqueName: \"kubernetes.io/projected/aa59b9bc-694f-4b16-99bf-5d31a85bca74-kube-api-access-k8bxh\") pod \"node-resolver-xbz24\" (UID: \"aa59b9bc-694f-4b16-99bf-5d31a85bca74\") " pod="openshift-dns/node-resolver-xbz24"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515909    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-sys-fs\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515931    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/077f92f7-ee39-40c4-9873-43b00c9141a9-konnectivity-ca\") pod \"konnectivity-agent-xmrrh\" (UID: \"077f92f7-ee39-40c4-9873-43b00c9141a9\") " pod="kube-system/konnectivity-agent-xmrrh"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515953    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60babd6b-456a-43cb-81c8-0967234b4c7a-cnibin\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.515977    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38e19c6c-afca-4121-a242-f497fddd6827-cni-binary-copy\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516001    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60babd6b-456a-43cb-81c8-0967234b4c7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516024    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9ac6087e-296b-4917-bee6-8540193b8719-kubelet-config\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516048    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516070    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-run-openvswitch\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516091    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-node-log\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516112    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-log-socket\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.516322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516133    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gljgw\" (UniqueName: \"kubernetes.io/projected/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-kube-api-access-gljgw\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.516854 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516152    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-sysctl-d\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.516854 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516175    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-var-lib-kubelet\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.516854 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516222    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aad288b6-deba-464d-917f-dcc47d794865-host\") pod \"node-ca-zmgsq\" (UID: \"aad288b6-deba-464d-917f-dcc47d794865\") " pod="openshift-image-registry/node-ca-zmgsq"
Sep 18 18:39:04.516854 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516265    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs\") pod \"network-metrics-daemon-dtwsf\" (UID: \"2d9a33e8-d5ff-482e-b822-5aa13b177a6f\") " pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:04.516854 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516311    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-modprobe-d\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.516854 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516341    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5czx\" (UniqueName: \"kubernetes.io/projected/aad288b6-deba-464d-917f-dcc47d794865-kube-api-access-f5czx\") pod \"node-ca-zmgsq\" (UID: \"aad288b6-deba-464d-917f-dcc47d794865\") " pod="openshift-image-registry/node-ca-zmgsq"
Sep 18 18:39:04.516854 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516371    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60babd6b-456a-43cb-81c8-0967234b4c7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.516854 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516397    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa59b9bc-694f-4b16-99bf-5d31a85bca74-tmp-dir\") pod \"node-resolver-xbz24\" (UID: \"aa59b9bc-694f-4b16-99bf-5d31a85bca74\") " pod="openshift-dns/node-resolver-xbz24"
Sep 18 18:39:04.516854 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516420    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-var-lib-cni-bin\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.516854 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.516442    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-var-lib-cni-multus\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.542802 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.542783    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lmfwg"
Sep 18 18:39:04.548123 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.548098    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lmfwg"
Sep 18 18:39:04.575520 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.575465    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-78.ec2.internal" event={"ID":"8aa0497c56ca99c73b6260acf272ac5b","Type":"ContainerStarted","Data":"69170c558b3bd4482ddd1c2ec2026018211ca988a1dc3e4fa5486214a7351c99"}
Sep 18 18:39:04.577430 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.577372    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal" event={"ID":"b9265269b110ff616d074999444471d7","Type":"ContainerStarted","Data":"b4f19ed7f71841ab8b411a32eb134df85ce7c0cb74c142d780427f6384cd0248"}
Sep 18 18:39:04.616643 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616616    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-log-socket\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.616643 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616654    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gljgw\" (UniqueName: \"kubernetes.io/projected/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-kube-api-access-gljgw\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.616832 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616678    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-sysctl-d\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.616832 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616703    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-var-lib-kubelet\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.616832 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616726    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aad288b6-deba-464d-917f-dcc47d794865-host\") pod \"node-ca-zmgsq\" (UID: \"aad288b6-deba-464d-917f-dcc47d794865\") " pod="openshift-image-registry/node-ca-zmgsq"
Sep 18 18:39:04.616832 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616748    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs\") pod \"network-metrics-daemon-dtwsf\" (UID: \"2d9a33e8-d5ff-482e-b822-5aa13b177a6f\") " pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:04.616832 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616747    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-log-socket\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.616832 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616771    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-modprobe-d\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.616832 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616795    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5czx\" (UniqueName: \"kubernetes.io/projected/aad288b6-deba-464d-917f-dcc47d794865-kube-api-access-f5czx\") pod \"node-ca-zmgsq\" (UID: \"aad288b6-deba-464d-917f-dcc47d794865\") " pod="openshift-image-registry/node-ca-zmgsq"
Sep 18 18:39:04.616832 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616819    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-var-lib-kubelet\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.616832 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616820    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60babd6b-456a-43cb-81c8-0967234b4c7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616853    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa59b9bc-694f-4b16-99bf-5d31a85bca74-tmp-dir\") pod \"node-resolver-xbz24\" (UID: \"aa59b9bc-694f-4b16-99bf-5d31a85bca74\") " pod="openshift-dns/node-resolver-xbz24"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-var-lib-cni-bin\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616904    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-var-lib-cni-multus\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616919    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4hl8\" (UniqueName: \"kubernetes.io/projected/38e19c6c-afca-4121-a242-f497fddd6827-kube-api-access-v4hl8\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616934    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-registration-dir\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616948    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60babd6b-456a-43cb-81c8-0967234b4c7a-system-cni-dir\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616967    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-sysctl-conf\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.616987    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-cnibin\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617010    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/38e19c6c-afca-4121-a242-f497fddd6827-multus-daemon-config\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617039    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-device-dir\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617065    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-var-lib-openvswitch\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617088    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-system-cni-dir\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617119    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrh7\" (UniqueName: \"kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7\") pod \"network-check-target-xrssq\" (UID: \"d2997254-b30e-438d-b105-40088a498ee5\") " pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617142    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-kubernetes\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617165    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-run-multus-certs\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617229    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-run-multus-certs\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.617284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617238    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60babd6b-456a-43cb-81c8-0967234b4c7a-system-cni-dir\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617282    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/077f92f7-ee39-40c4-9873-43b00c9141a9-agent-certs\") pod \"konnectivity-agent-xmrrh\" (UID: \"077f92f7-ee39-40c4-9873-43b00c9141a9\") " pod="kube-system/konnectivity-agent-xmrrh"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617311    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617337    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-ovn-node-metrics-cert\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617359    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-sysctl-conf\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617365    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-kubelet\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617393    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60babd6b-456a-43cb-81c8-0967234b4c7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617402    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-kubelet\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617416    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-ovnkube-config\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-env-overrides\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617468    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-run-netns\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617513    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk75h\" (UniqueName: \"kubernetes.io/projected/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-kube-api-access-qk75h\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617537    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-etc-openvswitch\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617563    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-ovnkube-script-lib\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617611    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-sys\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617643    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aad288b6-deba-464d-917f-dcc47d794865-serviceca\") pod \"node-ca-zmgsq\" (UID: \"aad288b6-deba-464d-917f-dcc47d794865\") " pod="openshift-image-registry/node-ca-zmgsq"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617666    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-run-k8s-cni-cncf-io\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617672    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aad288b6-deba-464d-917f-dcc47d794865-host\") pod \"node-ca-zmgsq\" (UID: \"aad288b6-deba-464d-917f-dcc47d794865\") " pod="openshift-image-registry/node-ca-zmgsq"
Sep 18 18:39:04.618038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617690    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-hostroot\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617734    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-multus-conf-dir\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:04.617759    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617758    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-etc-kubernetes\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617788    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617807    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa59b9bc-694f-4b16-99bf-5d31a85bca74-tmp-dir\") pod \"node-resolver-xbz24\" (UID: \"aa59b9bc-694f-4b16-99bf-5d31a85bca74\") " pod="openshift-dns/node-resolver-xbz24"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617860    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-socket-dir\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617894    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-multus-cni-dir\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617908    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-var-lib-cni-bin\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617919    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3295807d-bb8e-4948-9a6a-e438b0f4e048-tmp\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-multus-socket-dir-parent\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617951    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-var-lib-cni-multus\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617967    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-var-lib-kubelet\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618001    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-slash\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618023    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618039    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s892\" (UniqueName: \"kubernetes.io/projected/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-kube-api-access-7s892\") pod \"network-metrics-daemon-dtwsf\" (UID: \"2d9a33e8-d5ff-482e-b822-5aa13b177a6f\") " pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618070    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-tuned\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618073    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-modprobe-d\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.618936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618092    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21bb6420-c1bc-4591-b804-4a69aa5092d6-iptables-alerter-script\") pod \"iptables-alerter-l769f\" (UID: \"21bb6420-c1bc-4591-b804-4a69aa5092d6\") " pod="openshift-network-operator/iptables-alerter-l769f"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618115    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-systemd-units\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618149    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618170    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60babd6b-456a-43cb-81c8-0967234b4c7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618206    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-registration-dir\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618245    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-run-ovn\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618269    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-systemd\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618317    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz2wg\" (UniqueName: \"kubernetes.io/projected/3295807d-bb8e-4948-9a6a-e438b0f4e048-kube-api-access-kz2wg\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618360    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njxzv\" (UniqueName: \"kubernetes.io/projected/60babd6b-456a-43cb-81c8-0967234b4c7a-kube-api-access-njxzv\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618427    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-multus-conf-dir\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618430    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9ac6087e-296b-4917-bee6-8540193b8719-dbus\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618469    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-run-netns\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618480    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-cnibin\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618502    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-sysconfig\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618533    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-host\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618558    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618580    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21bb6420-c1bc-4591-b804-4a69aa5092d6-host-slash\") pod \"iptables-alerter-l769f\" (UID: \"21bb6420-c1bc-4591-b804-4a69aa5092d6\") " pod="openshift-network-operator/iptables-alerter-l769f"
Sep 18 18:39:04.619747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618624    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lwgk\" (UniqueName: \"kubernetes.io/projected/21bb6420-c1bc-4591-b804-4a69aa5092d6-kube-api-access-4lwgk\") pod \"iptables-alerter-l769f\" (UID: \"21bb6420-c1bc-4591-b804-4a69aa5092d6\") " pod="openshift-network-operator/iptables-alerter-l769f"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618647    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60babd6b-456a-43cb-81c8-0967234b4c7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618673    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-run-systemd\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618706    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-cni-bin\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618730    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-cni-netd\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618755    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-run\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618793    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-lib-modules\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618848    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa59b9bc-694f-4b16-99bf-5d31a85bca74-hosts-file\") pod \"node-resolver-xbz24\" (UID: \"aa59b9bc-694f-4b16-99bf-5d31a85bca74\") " pod="openshift-dns/node-resolver-xbz24"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618897    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bxh\" (UniqueName: \"kubernetes.io/projected/aa59b9bc-694f-4b16-99bf-5d31a85bca74-kube-api-access-k8bxh\") pod \"node-resolver-xbz24\" (UID: \"aa59b9bc-694f-4b16-99bf-5d31a85bca74\") " pod="openshift-dns/node-resolver-xbz24"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618939    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-sys-fs\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.618998    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/38e19c6c-afca-4121-a242-f497fddd6827-multus-daemon-config\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619033    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/077f92f7-ee39-40c4-9873-43b00c9141a9-konnectivity-ca\") pod \"konnectivity-agent-xmrrh\" (UID: \"077f92f7-ee39-40c4-9873-43b00c9141a9\") " pod="kube-system/konnectivity-agent-xmrrh"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619073    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60babd6b-456a-43cb-81c8-0967234b4c7a-cnibin\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619078    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-device-dir\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619097    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38e19c6c-afca-4121-a242-f497fddd6827-cni-binary-copy\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619122    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60babd6b-456a-43cb-81c8-0967234b4c7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619126    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-var-lib-openvswitch\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.620632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619146    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9ac6087e-296b-4917-bee6-8540193b8719-kubelet-config\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619230    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9ac6087e-296b-4917-bee6-8540193b8719-kubelet-config\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619294    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-etc-kubernetes\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619370    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619437    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-system-cni-dir\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619496    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-sys\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619549    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-socket-dir\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619554    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-etc-openvswitch\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.617538    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-sysctl-d\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619623    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-kubernetes\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619810    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-multus-cni-dir\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619876    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-sysconfig\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619930    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-multus-socket-dir-parent\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619974    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-run\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.619981    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21bb6420-c1bc-4591-b804-4a69aa5092d6-host-slash\") pod \"iptables-alerter-l769f\" (UID: \"21bb6420-c1bc-4591-b804-4a69aa5092d6\") " pod="openshift-network-operator/iptables-alerter-l769f"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620016    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-host\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620055    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-ovnkube-script-lib\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620059    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-var-lib-kubelet\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.621437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620099    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-slash\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620113    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-sys-fs\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620143    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620178    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-lib-modules\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620257    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa59b9bc-694f-4b16-99bf-5d31a85bca74-hosts-file\") pod \"node-resolver-xbz24\" (UID: \"aa59b9bc-694f-4b16-99bf-5d31a85bca74\") " pod="openshift-dns/node-resolver-xbz24"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620257    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-ovnkube-config\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620411    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-run-ovn\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620526    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60babd6b-456a-43cb-81c8-0967234b4c7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620708    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60babd6b-456a-43cb-81c8-0967234b4c7a-cnibin\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620713    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-env-overrides\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620784    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-run-netns\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620798    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9ac6087e-296b-4917-bee6-8540193b8719-dbus\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620558    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-systemd\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:04.619630    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs podName:2d9a33e8-d5ff-482e-b822-5aa13b177a6f nodeName:}" failed. No retries permitted until 2026-09-18 18:39:05.11956757 +0000 UTC m=+4.556146399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs") pod "network-metrics-daemon-dtwsf" (UID: "2d9a33e8-d5ff-482e-b822-5aa13b177a6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620861    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-run-netns\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620901    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-cni-bin\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620941    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-run-systemd\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.622269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620973    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-hostroot\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.620975    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-host-cni-netd\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.621049    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.621166    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-run-openvswitch\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.621216    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-node-log\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.621311    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.621390    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38e19c6c-afca-4121-a242-f497fddd6827-cni-binary-copy\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.621483    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/077f92f7-ee39-40c4-9873-43b00c9141a9-konnectivity-ca\") pod \"konnectivity-agent-xmrrh\" (UID: \"077f92f7-ee39-40c4-9873-43b00c9141a9\") " pod="kube-system/konnectivity-agent-xmrrh"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.621537    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/38e19c6c-afca-4121-a242-f497fddd6827-host-run-k8s-cni-cncf-io\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.621846    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60babd6b-456a-43cb-81c8-0967234b4c7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.621890    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21bb6420-c1bc-4591-b804-4a69aa5092d6-iptables-alerter-script\") pod \"iptables-alerter-l769f\" (UID: \"21bb6420-c1bc-4591-b804-4a69aa5092d6\") " pod="openshift-network-operator/iptables-alerter-l769f"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.621931    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60babd6b-456a-43cb-81c8-0967234b4c7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.621993    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-run-openvswitch\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.622032    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-systemd-units\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.622047    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-node-log\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.622047    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aad288b6-deba-464d-917f-dcc47d794865-serviceca\") pod \"node-ca-zmgsq\" (UID: \"aad288b6-deba-464d-917f-dcc47d794865\") " pod="openshift-image-registry/node-ca-zmgsq"
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:04.622105    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:39:04.623011 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:04.622174    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret podName:9ac6087e-296b-4917-bee6-8540193b8719 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:05.122156836 +0000 UTC m=+4.558735662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret") pod "global-pull-secret-syncer-w9bz8" (UID: "9ac6087e-296b-4917-bee6-8540193b8719") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:39:04.623670 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.622741    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3295807d-bb8e-4948-9a6a-e438b0f4e048-etc-tuned\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.623670 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.622771    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3295807d-bb8e-4948-9a6a-e438b0f4e048-tmp\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.623670 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.622946    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/077f92f7-ee39-40c4-9873-43b00c9141a9-agent-certs\") pod \"konnectivity-agent-xmrrh\" (UID: \"077f92f7-ee39-40c4-9873-43b00c9141a9\") " pod="kube-system/konnectivity-agent-xmrrh"
Sep 18 18:39:04.624477 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.624302    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-ovn-node-metrics-cert\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.626817 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:04.626031    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:39:04.626817 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:04.626058    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:39:04.626817 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:04.626073    2558 projected.go:196] Error preparing data for projected volume kube-api-access-znrh7 for pod openshift-network-diagnostics/network-check-target-xrssq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:39:04.626817 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:04.626122    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7 podName:d2997254-b30e-438d-b105-40088a498ee5 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:05.126106855 +0000 UTC m=+4.562685674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-znrh7" (UniqueName: "kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7") pod "network-check-target-xrssq" (UID: "d2997254-b30e-438d-b105-40088a498ee5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:39:04.626817 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.626714    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gljgw\" (UniqueName: \"kubernetes.io/projected/9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd-kube-api-access-gljgw\") pod \"ovnkube-node-l6vdn\" (UID: \"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.628269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.628173    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4hl8\" (UniqueName: \"kubernetes.io/projected/38e19c6c-afca-4121-a242-f497fddd6827-kube-api-access-v4hl8\") pod \"multus-4ww5h\" (UID: \"38e19c6c-afca-4121-a242-f497fddd6827\") " pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.629804 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.629662    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5czx\" (UniqueName: \"kubernetes.io/projected/aad288b6-deba-464d-917f-dcc47d794865-kube-api-access-f5czx\") pod \"node-ca-zmgsq\" (UID: \"aad288b6-deba-464d-917f-dcc47d794865\") " pod="openshift-image-registry/node-ca-zmgsq"
Sep 18 18:39:04.631978 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.631936    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8bxh\" (UniqueName: \"kubernetes.io/projected/aa59b9bc-694f-4b16-99bf-5d31a85bca74-kube-api-access-k8bxh\") pod \"node-resolver-xbz24\" (UID: \"aa59b9bc-694f-4b16-99bf-5d31a85bca74\") " pod="openshift-dns/node-resolver-xbz24"
Sep 18 18:39:04.632075 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.632055    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz2wg\" (UniqueName: \"kubernetes.io/projected/3295807d-bb8e-4948-9a6a-e438b0f4e048-kube-api-access-kz2wg\") pod \"tuned-7j542\" (UID: \"3295807d-bb8e-4948-9a6a-e438b0f4e048\") " pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.632386 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.632329    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s892\" (UniqueName: \"kubernetes.io/projected/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-kube-api-access-7s892\") pod \"network-metrics-daemon-dtwsf\" (UID: \"2d9a33e8-d5ff-482e-b822-5aa13b177a6f\") " pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:04.632553 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.632535    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njxzv\" (UniqueName: \"kubernetes.io/projected/60babd6b-456a-43cb-81c8-0967234b4c7a-kube-api-access-njxzv\") pod \"multus-additional-cni-plugins-thdgp\" (UID: \"60babd6b-456a-43cb-81c8-0967234b4c7a\") " pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.636234 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.635768    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lwgk\" (UniqueName: \"kubernetes.io/projected/21bb6420-c1bc-4591-b804-4a69aa5092d6-kube-api-access-4lwgk\") pod \"iptables-alerter-l769f\" (UID: \"21bb6420-c1bc-4591-b804-4a69aa5092d6\") " pod="openshift-network-operator/iptables-alerter-l769f"
Sep 18 18:39:04.636522 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.636502    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk75h\" (UniqueName: \"kubernetes.io/projected/a336fc46-68b8-4cb4-b0c4-5d45f898b6fa-kube-api-access-qk75h\") pod \"aws-ebs-csi-driver-node-qdsfc\" (UID: \"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.789831 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.789733    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l769f"
Sep 18 18:39:04.796985 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.796771    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc"
Sep 18 18:39:04.802849 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.802825    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xmrrh"
Sep 18 18:39:04.803769 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:04.803743    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21bb6420_c1bc_4591_b804_4a69aa5092d6.slice/crio-180e980703cb80bc02f8315241041e48f35c536d9a3f93034130e8419464e918 WatchSource:0}: Error finding container 180e980703cb80bc02f8315241041e48f35c536d9a3f93034130e8419464e918: Status 404 returned error can't find the container with id 180e980703cb80bc02f8315241041e48f35c536d9a3f93034130e8419464e918
Sep 18 18:39:04.808153 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.808135    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7j542"
Sep 18 18:39:04.814385 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.814238    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zmgsq"
Sep 18 18:39:04.816616 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:04.816585    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda336fc46_68b8_4cb4_b0c4_5d45f898b6fa.slice/crio-ad2314ce96e5394b1ea9548369da37978aa200c8ef9eeec3a139a0acce1551de WatchSource:0}: Error finding container ad2314ce96e5394b1ea9548369da37978aa200c8ef9eeec3a139a0acce1551de: Status 404 returned error can't find the container with id ad2314ce96e5394b1ea9548369da37978aa200c8ef9eeec3a139a0acce1551de
Sep 18 18:39:04.819896 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.819754    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xbz24"
Sep 18 18:39:04.820137 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:04.820113    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077f92f7_ee39_40c4_9873_43b00c9141a9.slice/crio-b04269f37194b5f1f572184e9685b0a5675af6462ccaf3407458547e4f598c03 WatchSource:0}: Error finding container b04269f37194b5f1f572184e9685b0a5675af6462ccaf3407458547e4f598c03: Status 404 returned error can't find the container with id b04269f37194b5f1f572184e9685b0a5675af6462ccaf3407458547e4f598c03
Sep 18 18:39:04.826142 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.826119    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-thdgp"
Sep 18 18:39:04.826638 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:04.826593    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3295807d_bb8e_4948_9a6a_e438b0f4e048.slice/crio-4791df0e9b4d44241b673c441a70ba4d0d89450d99b15e4df8ec4801d542efe0 WatchSource:0}: Error finding container 4791df0e9b4d44241b673c441a70ba4d0d89450d99b15e4df8ec4801d542efe0: Status 404 returned error can't find the container with id 4791df0e9b4d44241b673c441a70ba4d0d89450d99b15e4df8ec4801d542efe0
Sep 18 18:39:04.831120 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.831100    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4ww5h"
Sep 18 18:39:04.831811 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:04.831728    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa59b9bc_694f_4b16_99bf_5d31a85bca74.slice/crio-5d1cc03fd2188af3fcc78d9730b5c280706588e06277f912a1756c4dc53ea5bf WatchSource:0}: Error finding container 5d1cc03fd2188af3fcc78d9730b5c280706588e06277f912a1756c4dc53ea5bf: Status 404 returned error can't find the container with id 5d1cc03fd2188af3fcc78d9730b5c280706588e06277f912a1756c4dc53ea5bf
Sep 18 18:39:04.836835 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.836796    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:04.840901 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:04.840650    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60babd6b_456a_43cb_81c8_0967234b4c7a.slice/crio-49fbc2e58f7a70fce63171a29da2930f6b2f63389bbd7124dde4a01b0d42ebe4 WatchSource:0}: Error finding container 49fbc2e58f7a70fce63171a29da2930f6b2f63389bbd7124dde4a01b0d42ebe4: Status 404 returned error can't find the container with id 49fbc2e58f7a70fce63171a29da2930f6b2f63389bbd7124dde4a01b0d42ebe4
Sep 18 18:39:04.845813 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:04.845791    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e19c6c_afca_4121_a242_f497fddd6827.slice/crio-4142f4d9667dcc7eaaa78a4183c06c9546a2a50d547a528bf4e9ba5051a8fdd1 WatchSource:0}: Error finding container 4142f4d9667dcc7eaaa78a4183c06c9546a2a50d547a528bf4e9ba5051a8fdd1: Status 404 returned error can't find the container with id 4142f4d9667dcc7eaaa78a4183c06c9546a2a50d547a528bf4e9ba5051a8fdd1
Sep 18 18:39:04.869636 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:04.869615    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:39:05.125351 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:05.125304    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs\") pod \"network-metrics-daemon-dtwsf\" (UID: \"2d9a33e8-d5ff-482e-b822-5aa13b177a6f\") " pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:05.125574 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:05.125551    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:05.125684 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:05.125563    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:39:05.125684 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:05.125653    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs podName:2d9a33e8-d5ff-482e-b822-5aa13b177a6f nodeName:}" failed. No retries permitted until 2026-09-18 18:39:06.125633663 +0000 UTC m=+5.562212494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs") pod "network-metrics-daemon-dtwsf" (UID: "2d9a33e8-d5ff-482e-b822-5aa13b177a6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:39:05.125684 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:05.125663    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:39:05.125860 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:05.125718    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret podName:9ac6087e-296b-4917-bee6-8540193b8719 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:06.125705369 +0000 UTC m=+5.562284192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret") pod "global-pull-secret-syncer-w9bz8" (UID: "9ac6087e-296b-4917-bee6-8540193b8719") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:39:05.226487 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:05.226454    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrh7\" (UniqueName: \"kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7\") pod \"network-check-target-xrssq\" (UID: \"d2997254-b30e-438d-b105-40088a498ee5\") " pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:05.226672 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:05.226650    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:39:05.226672 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:05.226670    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:39:05.226781 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:05.226683    2558 projected.go:196] Error preparing data for projected volume kube-api-access-znrh7 for pod openshift-network-diagnostics/network-check-target-xrssq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:39:05.226781 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:05.226759    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7 podName:d2997254-b30e-438d-b105-40088a498ee5 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:06.226732375 +0000 UTC m=+5.663311199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-znrh7" (UniqueName: "kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7") pod "network-check-target-xrssq" (UID: "d2997254-b30e-438d-b105-40088a498ee5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:39:05.550809 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:05.550717    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 18:34:04 +0000 UTC" deadline="2028-05-18 15:32:15.813395859 +0000 UTC"
Sep 18 18:39:05.550809 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:05.550749    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14588h53m10.262651357s"
Sep 18 18:39:05.580601 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:05.580565    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn" event={"ID":"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd","Type":"ContainerStarted","Data":"f9bf44b78f9219db33f9cf9acf205f35ae173538ac1167f9dd111da0c062d84e"}
Sep 18 18:39:05.582005 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:05.581975    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xbz24" event={"ID":"aa59b9bc-694f-4b16-99bf-5d31a85bca74","Type":"ContainerStarted","Data":"5d1cc03fd2188af3fcc78d9730b5c280706588e06277f912a1756c4dc53ea5bf"}
Sep 18 18:39:05.583129 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:05.583101    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zmgsq" event={"ID":"aad288b6-deba-464d-917f-dcc47d794865","Type":"ContainerStarted","Data":"c4327e0c52cdbbfd9ac3c7b0e08e6930d3a1ea62f23e23aa0ee25df04128ff5a"}
Sep 18 18:39:05.584484 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:05.584458    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7j542" event={"ID":"3295807d-bb8e-4948-9a6a-e438b0f4e048","Type":"ContainerStarted","Data":"4791df0e9b4d44241b673c441a70ba4d0d89450d99b15e4df8ec4801d542efe0"}
Sep 18 18:39:05.585780 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:05.585754    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc" event={"ID":"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa","Type":"ContainerStarted","Data":"ad2314ce96e5394b1ea9548369da37978aa200c8ef9eeec3a139a0acce1551de"}
Sep 18 18:39:05.586837 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:05.586813    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l769f" event={"ID":"21bb6420-c1bc-4591-b804-4a69aa5092d6","Type":"ContainerStarted","Data":"180e980703cb80bc02f8315241041e48f35c536d9a3f93034130e8419464e918"}
Sep 18 18:39:05.587884 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:05.587860    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4ww5h" event={"ID":"38e19c6c-afca-4121-a242-f497fddd6827","Type":"ContainerStarted","Data":"4142f4d9667dcc7eaaa78a4183c06c9546a2a50d547a528bf4e9ba5051a8fdd1"}
Sep 18 18:39:05.588897 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:05.588879    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thdgp" event={"ID":"60babd6b-456a-43cb-81c8-0967234b4c7a","Type":"ContainerStarted","Data":"49fbc2e58f7a70fce63171a29da2930f6b2f63389bbd7124dde4a01b0d42ebe4"}
Sep 18 18:39:05.590032 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:05.590002    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xmrrh" event={"ID":"077f92f7-ee39-40c4-9873-43b00c9141a9","Type":"ContainerStarted","Data":"b04269f37194b5f1f572184e9685b0a5675af6462ccaf3407458547e4f598c03"}
Sep 18 18:39:06.133002 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:06.132936    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:06.133213 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:06.133020    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs\") pod \"network-metrics-daemon-dtwsf\" (UID: \"2d9a33e8-d5ff-482e-b822-5aa13b177a6f\") " pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:06.133287 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:06.133272    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:39:06.133876 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:06.133337    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs podName:2d9a33e8-d5ff-482e-b822-5aa13b177a6f nodeName:}" failed. No retries permitted until 2026-09-18 18:39:08.13331746 +0000 UTC m=+7.569896301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs") pod "network-metrics-daemon-dtwsf" (UID: "2d9a33e8-d5ff-482e-b822-5aa13b177a6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:39:06.133876 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:06.133750    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:39:06.133876 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:06.133806    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret podName:9ac6087e-296b-4917-bee6-8540193b8719 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:08.13378152 +0000 UTC m=+7.570360343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret") pod "global-pull-secret-syncer-w9bz8" (UID: "9ac6087e-296b-4917-bee6-8540193b8719") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:39:06.234351 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:06.234248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrh7\" (UniqueName: \"kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7\") pod \"network-check-target-xrssq\" (UID: \"d2997254-b30e-438d-b105-40088a498ee5\") " pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:06.234514 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:06.234388    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:39:06.234514 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:06.234412    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:39:06.234514 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:06.234424    2558 projected.go:196] Error preparing data for projected volume kube-api-access-znrh7 for pod openshift-network-diagnostics/network-check-target-xrssq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:39:06.234514 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:06.234487    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7 podName:d2997254-b30e-438d-b105-40088a498ee5 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:08.234470194 +0000 UTC m=+7.671049031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-znrh7" (UniqueName: "kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7") pod "network-check-target-xrssq" (UID: "d2997254-b30e-438d-b105-40088a498ee5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:39:06.551420 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:06.551361    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 18:34:04 +0000 UTC" deadline="2028-03-08 00:28:53.183697893 +0000 UTC"
Sep 18 18:39:06.551420 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:06.551421    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12869h49m46.632280982s"
Sep 18 18:39:06.570473 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:06.569752    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:06.570473 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:06.569880    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:06.570473 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:06.570315    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:06.570473 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:06.570426    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:06.570969 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:06.570842    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:06.570969 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:06.570930    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:07.612410 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:07.612150    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-78.ec2.internal" event={"ID":"8aa0497c56ca99c73b6260acf272ac5b","Type":"ContainerStarted","Data":"dcd4f131664ffd1b8c116f88800c0819320fdba3ce1191e73110d4815504a814"}
Sep 18 18:39:07.618315 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:07.617708    2558 generic.go:356] "Generic (PLEG): container finished" podID="b9265269b110ff616d074999444471d7" containerID="5a1cfa684f1bb56a390b7223be0a4108564d54fb44b771b1d958b3b99a618717" exitCode=0
Sep 18 18:39:07.618315 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:07.617753    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal" event={"ID":"b9265269b110ff616d074999444471d7","Type":"ContainerDied","Data":"5a1cfa684f1bb56a390b7223be0a4108564d54fb44b771b1d958b3b99a618717"}
Sep 18 18:39:07.628142 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:07.628078    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-78.ec2.internal" podStartSLOduration=4.628059185 podStartE2EDuration="4.628059185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:07.627085104 +0000 UTC m=+7.063663946" watchObservedRunningTime="2026-09-18 18:39:07.628059185 +0000 UTC m=+7.064638028"
Sep 18 18:39:08.150056 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:08.149343    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:08.150056 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:08.149395    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs\") pod \"network-metrics-daemon-dtwsf\" (UID: \"2d9a33e8-d5ff-482e-b822-5aa13b177a6f\") " pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:08.150056 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:08.149531    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:39:08.150056 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:08.149593    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs podName:2d9a33e8-d5ff-482e-b822-5aa13b177a6f nodeName:}" failed. No retries permitted until 2026-09-18 18:39:12.149574968 +0000 UTC m=+11.586153810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs") pod "network-metrics-daemon-dtwsf" (UID: "2d9a33e8-d5ff-482e-b822-5aa13b177a6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:39:08.150056 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:08.149970    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:39:08.150056 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:08.150013    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret podName:9ac6087e-296b-4917-bee6-8540193b8719 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:12.149999448 +0000 UTC m=+11.586578271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret") pod "global-pull-secret-syncer-w9bz8" (UID: "9ac6087e-296b-4917-bee6-8540193b8719") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:39:08.250155 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:08.250118    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrh7\" (UniqueName: \"kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7\") pod \"network-check-target-xrssq\" (UID: \"d2997254-b30e-438d-b105-40088a498ee5\") " pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:08.250359 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:08.250340    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:39:08.250418 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:08.250361    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:39:08.250418 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:08.250376    2558 projected.go:196] Error preparing data for projected volume kube-api-access-znrh7 for pod openshift-network-diagnostics/network-check-target-xrssq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:39:08.250515 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:08.250434    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7 podName:d2997254-b30e-438d-b105-40088a498ee5 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:12.250418067 +0000 UTC m=+11.686996892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-znrh7" (UniqueName: "kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7") pod "network-check-target-xrssq" (UID: "d2997254-b30e-438d-b105-40088a498ee5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:39:08.569459 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:08.569375    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:08.569459 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:08.569411    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:08.569682 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:08.569377    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:08.569682 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:08.569527    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:08.569682 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:08.569616    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:08.569824 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:08.569697    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:10.569905 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:10.569206    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:10.569905 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:10.569377    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:10.569905 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:10.569827    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:10.569905 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:10.569835    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:10.570566 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:10.569918    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:10.570566 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:10.570005    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:12.183670 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:12.183630    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:12.184129 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:12.183679    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs\") pod \"network-metrics-daemon-dtwsf\" (UID: \"2d9a33e8-d5ff-482e-b822-5aa13b177a6f\") " pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:12.184129 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:12.183801    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:39:12.184129 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:12.183801    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:39:12.184129 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:12.183860    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs podName:2d9a33e8-d5ff-482e-b822-5aa13b177a6f nodeName:}" failed. No retries permitted until 2026-09-18 18:39:20.183840003 +0000 UTC m=+19.620418829 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs") pod "network-metrics-daemon-dtwsf" (UID: "2d9a33e8-d5ff-482e-b822-5aa13b177a6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:39:12.184129 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:12.183880    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret podName:9ac6087e-296b-4917-bee6-8540193b8719 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:20.183869469 +0000 UTC m=+19.620448289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret") pod "global-pull-secret-syncer-w9bz8" (UID: "9ac6087e-296b-4917-bee6-8540193b8719") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:39:12.284397 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:12.284322    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrh7\" (UniqueName: \"kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7\") pod \"network-check-target-xrssq\" (UID: \"d2997254-b30e-438d-b105-40088a498ee5\") " pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:12.284572 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:12.284499    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:39:12.284572 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:12.284527    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:39:12.284572 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:12.284539    2558 projected.go:196] Error preparing data for projected volume kube-api-access-znrh7 for pod openshift-network-diagnostics/network-check-target-xrssq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:39:12.284729 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:12.284599    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7 podName:d2997254-b30e-438d-b105-40088a498ee5 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:20.284584015 +0000 UTC m=+19.721162901 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-znrh7" (UniqueName: "kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7") pod "network-check-target-xrssq" (UID: "d2997254-b30e-438d-b105-40088a498ee5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:39:12.570588 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:12.570558    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:12.570757 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:12.570685    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:12.570757 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:12.570750    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:12.570868 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:12.570840    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:12.570922 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:12.570877    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:12.570967 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:12.570939    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:14.569113 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:14.569073    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:14.569583 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:14.569263    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:14.569583 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:14.569352    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:14.569822 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:14.569760    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:14.569946 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:14.569868    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:14.570022 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:14.569986    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:16.569254 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:16.569219    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:16.569254 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:16.569242    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:16.569793 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:16.569382    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:16.569793 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:16.569462    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:16.569793 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:16.569484    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:16.569793 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:16.569570    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:18.569704 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:18.569244    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:18.569704 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:18.569243    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:18.569704 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:18.569265    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:18.569704 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:18.569410    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:18.569704 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:18.569531    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:18.569704 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:18.569630    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:20.243067 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:20.243033    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:20.243067 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:20.243072    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs\") pod \"network-metrics-daemon-dtwsf\" (UID: \"2d9a33e8-d5ff-482e-b822-5aa13b177a6f\") " pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:20.243603 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:20.243204    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:39:20.243603 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:20.243261    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs podName:2d9a33e8-d5ff-482e-b822-5aa13b177a6f nodeName:}" failed. No retries permitted until 2026-09-18 18:39:36.243246595 +0000 UTC m=+35.679825434 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs") pod "network-metrics-daemon-dtwsf" (UID: "2d9a33e8-d5ff-482e-b822-5aa13b177a6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:39:20.243603 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:20.243201    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:39:20.243603 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:20.243293    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret podName:9ac6087e-296b-4917-bee6-8540193b8719 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:36.243286483 +0000 UTC m=+35.679865317 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret") pod "global-pull-secret-syncer-w9bz8" (UID: "9ac6087e-296b-4917-bee6-8540193b8719") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:39:20.344336 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:20.344300    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrh7\" (UniqueName: \"kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7\") pod \"network-check-target-xrssq\" (UID: \"d2997254-b30e-438d-b105-40088a498ee5\") " pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:20.344502 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:20.344453    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:39:20.344502 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:20.344474    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:39:20.344502 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:20.344486    2558 projected.go:196] Error preparing data for projected volume kube-api-access-znrh7 for pod openshift-network-diagnostics/network-check-target-xrssq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:39:20.344636 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:20.344548    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7 podName:d2997254-b30e-438d-b105-40088a498ee5 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:36.344524004 +0000 UTC m=+35.781102837 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-znrh7" (UniqueName: "kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7") pod "network-check-target-xrssq" (UID: "d2997254-b30e-438d-b105-40088a498ee5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:39:20.569203 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:20.569112    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:20.569203 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:20.569130    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:20.569399 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:20.569298    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:20.569399 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:20.569350    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:20.569497 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:20.569449    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:20.569605 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:20.569554    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:22.570283 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:22.570254    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:22.570646 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:22.570330    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:22.570646 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:22.570406    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:22.570646 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:22.570500    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:22.570646 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:22.570538    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:22.570646 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:22.570594    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:23.660602 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.660154    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal" event={"ID":"b9265269b110ff616d074999444471d7","Type":"ContainerStarted","Data":"a56b123de6ac9a42ef6199204acc4cbead788c33c9f63cf29a3ef65ed951f77b"}
Sep 18 18:39:23.663893 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.663869    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 18:39:23.664373 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.664281    2558 generic.go:356] "Generic (PLEG): container finished" podID="9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd" containerID="404153cd352ea0aa0a5974955e40fc1dff79c0b47a1c19eafcb254ee66216703" exitCode=1
Sep 18 18:39:23.664373 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.664350    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn" event={"ID":"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd","Type":"ContainerStarted","Data":"94a28f52ef1698825f2967398ccbd623b2d8f855c5193cd4f71ed9fbdd1a7e7f"}
Sep 18 18:39:23.664527 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.664426    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn" event={"ID":"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd","Type":"ContainerStarted","Data":"a1b4b8b9b2f0cbba7d325751b7fc8a60bc9d89cba7bb4f83b9af8858990894a5"}
Sep 18 18:39:23.664527 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.664475    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn" event={"ID":"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd","Type":"ContainerStarted","Data":"2c65ea8a701259fb4dcc66eae032a0ca57556c853a8c30b4995e59d2dc27cf2d"}
Sep 18 18:39:23.664527 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.664504    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn" event={"ID":"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd","Type":"ContainerStarted","Data":"e4fc5f96dfab066c3fc5a9153b978511dd6037ac75d433c9ef1103279ed9f9f0"}
Sep 18 18:39:23.664682 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.664530    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn" event={"ID":"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd","Type":"ContainerDied","Data":"404153cd352ea0aa0a5974955e40fc1dff79c0b47a1c19eafcb254ee66216703"}
Sep 18 18:39:23.664682 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.664558    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn" event={"ID":"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd","Type":"ContainerStarted","Data":"ef71960db788ec3dea7d2d454f78da74809d7bac6ade84ace4e1dc45b43e673b"}
Sep 18 18:39:23.666031 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.665986    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xbz24" event={"ID":"aa59b9bc-694f-4b16-99bf-5d31a85bca74","Type":"ContainerStarted","Data":"7dba1ed7cdfa53bddd2291e3b316de0fd9ad31aabc285ca2f5accc684a388dbe"}
Sep 18 18:39:23.667579 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.667544    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zmgsq" event={"ID":"aad288b6-deba-464d-917f-dcc47d794865","Type":"ContainerStarted","Data":"40c97d618e5fa8ed4c1a3e6d6a96101d69afc540430edf7f20caf5c1c91f113e"}
Sep 18 18:39:23.669351 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.669329    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7j542" event={"ID":"3295807d-bb8e-4948-9a6a-e438b0f4e048","Type":"ContainerStarted","Data":"45f0cc385149af0a57cf3c9848ffd9166f2856f9b021e4108f6bb1b355aa56db"}
Sep 18 18:39:23.671375 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.671325    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc" event={"ID":"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa","Type":"ContainerStarted","Data":"15a16d1c82aaf92b6a8b2c70af5f86f974e8215361385f4b012ee1d0409e5b63"}
Sep 18 18:39:23.673068 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.673045    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4ww5h" event={"ID":"38e19c6c-afca-4121-a242-f497fddd6827","Type":"ContainerStarted","Data":"f496fe18ea7168fc9f27f5077ca027a1ccc12571307d884e83120256a5e68bbc"}
Sep 18 18:39:23.675119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.675096    2558 generic.go:356] "Generic (PLEG): container finished" podID="60babd6b-456a-43cb-81c8-0967234b4c7a" containerID="4f81105190b751270a006ae4bd4055e9dc27151077a265934ea9c4315ceeb248" exitCode=0
Sep 18 18:39:23.675119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.675103    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thdgp" event={"ID":"60babd6b-456a-43cb-81c8-0967234b4c7a","Type":"ContainerDied","Data":"4f81105190b751270a006ae4bd4055e9dc27151077a265934ea9c4315ceeb248"}
Sep 18 18:39:23.677103 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.677073    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xmrrh" event={"ID":"077f92f7-ee39-40c4-9873-43b00c9141a9","Type":"ContainerStarted","Data":"80798cf48cc0be687fc1b842f6904fac4898719ec37a1dd828818c8d10c48e2b"}
Sep 18 18:39:23.690485 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.690448    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7j542" podStartSLOduration=3.705386973 podStartE2EDuration="21.690434404s" totalImagesPullingTime="17.985047431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:23.690080499 +0000 UTC m=+23.126659356" watchObservedRunningTime="2026-09-18 18:39:23.690434404 +0000 UTC m=+23.127013246"
Sep 18 18:39:23.690628 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.690594    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-78.ec2.internal" podStartSLOduration=20.690588675 podStartE2EDuration="20.690588675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:23.674590614 +0000 UTC m=+23.111169455" watchObservedRunningTime="2026-09-18 18:39:23.690588675 +0000 UTC m=+23.127167521"
Sep 18 18:39:23.707078 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.707036    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4ww5h" podStartSLOduration=3.628461411 podStartE2EDuration="21.707021706s" totalImagesPullingTime="18.078560295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:23.706717547 +0000 UTC m=+23.143296394" watchObservedRunningTime="2026-09-18 18:39:23.707021706 +0000 UTC m=+23.143600549"
Sep 18 18:39:23.751572 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.751523    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xbz24" podStartSLOduration=3.7761730289999997 podStartE2EDuration="21.751503413s" totalImagesPullingTime="17.975330384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:23.751024286 +0000 UTC m=+23.187603127" watchObservedRunningTime="2026-09-18 18:39:23.751503413 +0000 UTC m=+23.188082256"
Sep 18 18:39:23.751728 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.751597    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zmgsq" podStartSLOduration=3.716415638 podStartE2EDuration="21.751592039s" totalImagesPullingTime="18.035176401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:23.727211479 +0000 UTC m=+23.163790317" watchObservedRunningTime="2026-09-18 18:39:23.751592039 +0000 UTC m=+23.188170880"
Sep 18 18:39:23.764801 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:23.764756    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xmrrh" podStartSLOduration=7.892874619 podStartE2EDuration="21.764741831s" totalImagesPullingTime="13.871867212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:23.764495007 +0000 UTC m=+23.201073849" watchObservedRunningTime="2026-09-18 18:39:23.764741831 +0000 UTC m=+23.201320674"
Sep 18 18:39:24.524898 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:24.524871    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 18:39:24.569601 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:24.569575    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:24.569739 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:24.569572    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:24.569739 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:24.569712    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:24.569856 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:24.569750    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:24.569856 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:24.569826    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:24.569946 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:24.569901    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:24.589680 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:24.589608    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T18:39:24.524892508Z","UUID":"84225a21-3fb5-497d-b65c-aa19ac776b28","Handler":null,"Name":"","Endpoint":""}
Sep 18 18:39:24.591782 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:24.591764    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 18:39:24.591877 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:24.591792    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 18:39:24.681735 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:24.681707    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc" event={"ID":"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa","Type":"ContainerStarted","Data":"7c0beadd2021ad053c289f60006f9363d6615c24263a2fc52ddaff3536952319"}
Sep 18 18:39:24.683435 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:24.683399    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l769f" event={"ID":"21bb6420-c1bc-4591-b804-4a69aa5092d6","Type":"ContainerStarted","Data":"6ac58933402a615a101ed9f6abb3b77bfb95f5be7fa72d57af9f00863b5386f3"}
Sep 18 18:39:25.848420 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:25.848358    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xmrrh"
Sep 18 18:39:25.849115 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:25.849056    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xmrrh"
Sep 18 18:39:25.864349 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:25.864295    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l769f" podStartSLOduration=5.857098425 podStartE2EDuration="23.864271356s" totalImagesPullingTime="18.007172931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:24.697989424 +0000 UTC m=+24.134568265" watchObservedRunningTime="2026-09-18 18:39:25.864271356 +0000 UTC m=+25.300850199"
Sep 18 18:39:26.569795 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:26.569759    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:26.569982 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:26.569767    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:26.569982 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:26.569865    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:26.569982 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:26.569873    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:26.569982 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:26.569936    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:26.570219 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:26.570008    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:26.691660 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:26.691631    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 18:39:26.692075 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:26.692045    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn" event={"ID":"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd","Type":"ContainerStarted","Data":"65c012aab2f677fe8f7832c6b46e7c69416f0490809df3a75ed524b6a8702af8"}
Sep 18 18:39:26.694369 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:26.694336    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc" event={"ID":"a336fc46-68b8-4cb4-b0c4-5d45f898b6fa","Type":"ContainerStarted","Data":"27d94be9ebec07411f279e1a0826fd0a2e1377534aa549de0ff5850d4d51a276"}
Sep 18 18:39:26.718354 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:26.718315    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdsfc" podStartSLOduration=4.271391974 podStartE2EDuration="24.718302363s" totalImagesPullingTime="20.446910389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:39:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:26.717989855 +0000 UTC m=+26.154568698" watchObservedRunningTime="2026-09-18 18:39:26.718302363 +0000 UTC m=+26.154881204"
Sep 18 18:39:28.569746 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:28.569505    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:28.570169 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:28.569517    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:28.570169 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:28.569860    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:28.570169 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:28.569579    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:28.570169 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:28.569929    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:28.570169 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:28.570016    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:29.702894 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:29.702859    2558 generic.go:356] "Generic (PLEG): container finished" podID="60babd6b-456a-43cb-81c8-0967234b4c7a" containerID="5339d44a0543c00dd0a80831543ff02dc18b9be85082eccf53e6d2d2fc4cbb28" exitCode=0
Sep 18 18:39:29.703682 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:29.702944    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thdgp" event={"ID":"60babd6b-456a-43cb-81c8-0967234b4c7a","Type":"ContainerDied","Data":"5339d44a0543c00dd0a80831543ff02dc18b9be85082eccf53e6d2d2fc4cbb28"}
Sep 18 18:39:29.706677 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:29.706662    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 18:39:29.707020 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:29.707000    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn" event={"ID":"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd","Type":"ContainerStarted","Data":"810ec6fe262e402bf0d2fff042116fef4591b49a275495774e7c22b82a10f7c7"}
Sep 18 18:39:29.707239 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:29.707219    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:29.707312 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:29.707297    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:29.707490 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:29.707476    2558 scope.go:117] "RemoveContainer" containerID="404153cd352ea0aa0a5974955e40fc1dff79c0b47a1c19eafcb254ee66216703"
Sep 18 18:39:29.721905 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:29.721886    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:30.573053 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:30.572859    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:30.573242 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:30.572891    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:30.573242 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:30.573138    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:30.573346 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:30.573256    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:30.573346 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:30.572916    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:30.573444 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:30.573389    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:30.712709 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:30.712687    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 18:39:30.713084 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:30.713024    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn" event={"ID":"9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd","Type":"ContainerStarted","Data":"95b89370301478b664d6523d503fb8bd2bf4a81b480329dfdf3a3ed4b1a6ab60"}
Sep 18 18:39:30.713393 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:30.713374    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:30.728112 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:30.728088    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:39:30.742442 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:30.742392    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn" podStartSLOduration=9.707999004 podStartE2EDuration="27.742378863s" totalImagesPullingTime="18.034379859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:30.7411327 +0000 UTC m=+30.177711542" watchObservedRunningTime="2026-09-18 18:39:30.742378863 +0000 UTC m=+30.178957704"
Sep 18 18:39:30.788431 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:30.788394    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xmrrh"
Sep 18 18:39:30.788579 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:30.788503    2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 18:39:30.788994 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:30.788974    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xmrrh"
Sep 18 18:39:31.089226 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:31.089164    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xrssq"]
Sep 18 18:39:31.089347 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:31.089313    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:31.089448 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:31.089422    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:31.093220 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:31.093147    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w9bz8"]
Sep 18 18:39:31.093352 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:31.093332    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:31.093485 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:31.093445    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:31.093981 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:31.093953    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dtwsf"]
Sep 18 18:39:31.094072 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:31.094055    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:31.094212 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:31.094162    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:31.717403 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:31.717368    2558 generic.go:356] "Generic (PLEG): container finished" podID="60babd6b-456a-43cb-81c8-0967234b4c7a" containerID="8afd5a0e09adc7c534cec34dc2bda1c572ff24e96745adfe967c5d0784494aec" exitCode=0
Sep 18 18:39:31.717761 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:31.717449    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thdgp" event={"ID":"60babd6b-456a-43cb-81c8-0967234b4c7a","Type":"ContainerDied","Data":"8afd5a0e09adc7c534cec34dc2bda1c572ff24e96745adfe967c5d0784494aec"}
Sep 18 18:39:32.570079 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:32.570016    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:32.570244 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:32.570152    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:32.570244 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:32.570165    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:32.570366 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:32.570284    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:32.570366 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:32.570322    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:32.570366 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:32.570357    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:33.724272 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:33.724235    2558 generic.go:356] "Generic (PLEG): container finished" podID="60babd6b-456a-43cb-81c8-0967234b4c7a" containerID="cf01d43f5fd6eb31f2de25d0bb99b657b90c324da160cc480e6a48d966b91c03" exitCode=0
Sep 18 18:39:33.724646 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:33.724292    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thdgp" event={"ID":"60babd6b-456a-43cb-81c8-0967234b4c7a","Type":"ContainerDied","Data":"cf01d43f5fd6eb31f2de25d0bb99b657b90c324da160cc480e6a48d966b91c03"}
Sep 18 18:39:34.569166 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:34.569134    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:34.569166 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:34.569134    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:34.569393 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:34.569265    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:34.569393 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:34.569286    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:34.569393 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:34.569343    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:34.569558 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:34.569439    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:36.259520 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.259446    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:36.259520 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.259485    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs\") pod \"network-metrics-daemon-dtwsf\" (UID: \"2d9a33e8-d5ff-482e-b822-5aa13b177a6f\") " pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:36.259906 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.259579    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:39:36.259906 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.259586    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:39:36.259906 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.259635    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs podName:2d9a33e8-d5ff-482e-b822-5aa13b177a6f nodeName:}" failed. No retries permitted until 2026-09-18 18:40:08.259621867 +0000 UTC m=+67.696200692 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs") pod "network-metrics-daemon-dtwsf" (UID: "2d9a33e8-d5ff-482e-b822-5aa13b177a6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:39:36.259906 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.259648    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret podName:9ac6087e-296b-4917-bee6-8540193b8719 nodeName:}" failed. No retries permitted until 2026-09-18 18:40:08.259642703 +0000 UTC m=+67.696221523 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret") pod "global-pull-secret-syncer-w9bz8" (UID: "9ac6087e-296b-4917-bee6-8540193b8719") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:39:36.360616 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.360585    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrh7\" (UniqueName: \"kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7\") pod \"network-check-target-xrssq\" (UID: \"d2997254-b30e-438d-b105-40088a498ee5\") " pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:36.360769 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.360701    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:39:36.360769 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.360713    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:39:36.360769 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.360722    2558 projected.go:196] Error preparing data for projected volume kube-api-access-znrh7 for pod openshift-network-diagnostics/network-check-target-xrssq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:39:36.360769 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.360765    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7 podName:d2997254-b30e-438d-b105-40088a498ee5 nodeName:}" failed. No retries permitted until 2026-09-18 18:40:08.360753855 +0000 UTC m=+67.797332675 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-znrh7" (UniqueName: "kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7") pod "network-check-target-xrssq" (UID: "d2997254-b30e-438d-b105-40088a498ee5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:39:36.569142 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.569112    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:36.569302 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.569112    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:36.569302 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.569237    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtwsf" podUID="2d9a33e8-d5ff-482e-b822-5aa13b177a6f"
Sep 18 18:39:36.569302 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.569123    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:36.569427 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.569313    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w9bz8" podUID="9ac6087e-296b-4917-bee6-8540193b8719"
Sep 18 18:39:36.569427 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.569352    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrssq" podUID="d2997254-b30e-438d-b105-40088a498ee5"
Sep 18 18:39:36.660252 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.660223    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-78.ec2.internal" event="NodeReady"
Sep 18 18:39:36.660437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.660361    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 18:39:36.694267 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.694236    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bdbd487b5-zx747"]
Sep 18 18:39:36.713269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.711354    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bdbd487b5-zx747"]
Sep 18 18:39:36.713269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.711472    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.713269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.712634    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zw2cl"]
Sep 18 18:39:36.713903 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.713854    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 18:39:36.714024 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.713999    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 18:39:36.714147 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.714089    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dvnsn\""
Sep 18 18:39:36.714147 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.714130    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 18:39:36.720058 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.720035    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 18:39:36.727569 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.727551    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-42z88"]
Sep 18 18:39:36.728314 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.728287    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:36.732130 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.732109    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 18:39:36.732651 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.732629    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r76nj\""
Sep 18 18:39:36.735141 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.732752    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 18:39:36.746132 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.746104    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zw2cl"]
Sep 18 18:39:36.746280 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.746147    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-42z88"]
Sep 18 18:39:36.746280 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.746254    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-42z88"
Sep 18 18:39:36.748512 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.748468    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 18:39:36.748616 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.748534    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ljbnj\""
Sep 18 18:39:36.748616 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.748549    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 18:39:36.748616 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.748534    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 18:39:36.858690 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.858655    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xmrrh_077f92f7-ee39-40c4-9873-43b00c9141a9/konnectivity-agent/0.log" limit=1024
Sep 18 18:39:36.864441 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.864418    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-certificates\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.864565 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.864464    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:36.864565 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.864481    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86xlg\" (UniqueName: \"kubernetes.io/projected/6455458b-83a0-4f35-857a-f11c3866299e-kube-api-access-86xlg\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:36.864565 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.864542    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-bound-sa-token\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.864674 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.864565    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6455458b-83a0-4f35-857a-f11c3866299e-config-volume\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:36.864674 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.864638    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r8gv\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-kube-api-access-9r8gv\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.864674 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.864655    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf18e0b9-6a23-4338-893f-e8d7ca463c1e-cert\") pod \"ingress-canary-42z88\" (UID: \"bf18e0b9-6a23-4338-893f-e8d7ca463c1e\") " pod="openshift-ingress-canary/ingress-canary-42z88"
Sep 18 18:39:36.864783 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.864674    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.864783 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.864696    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6455458b-83a0-4f35-857a-f11c3866299e-tmp-dir\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:36.864783 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.864711    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s5hn\" (UniqueName: \"kubernetes.io/projected/bf18e0b9-6a23-4338-893f-e8d7ca463c1e-kube-api-access-6s5hn\") pod \"ingress-canary-42z88\" (UID: \"bf18e0b9-6a23-4338-893f-e8d7ca463c1e\") " pod="openshift-ingress-canary/ingress-canary-42z88"
Sep 18 18:39:36.864783 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.864738    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-image-registry-private-configuration\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.864783 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.864763    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-trusted-ca\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.864978 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.864789    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-installation-pull-secrets\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.864978 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.864848    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-ca-trust-extracted\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.947364 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.947323    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xmrrh_077f92f7-ee39-40c4-9873-43b00c9141a9/konnectivity-agent/0.log" limit=1024
Sep 18 18:39:36.965395 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.965374    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r8gv\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-kube-api-access-9r8gv\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.965504 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.965397    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf18e0b9-6a23-4338-893f-e8d7ca463c1e-cert\") pod \"ingress-canary-42z88\" (UID: \"bf18e0b9-6a23-4338-893f-e8d7ca463c1e\") " pod="openshift-ingress-canary/ingress-canary-42z88"
Sep 18 18:39:36.965504 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.965418    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.965504 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.965440    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6455458b-83a0-4f35-857a-f11c3866299e-tmp-dir\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:36.965648 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.965532    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 18:39:36.965648 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.965552    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdbd487b5-zx747: secret "image-registry-tls" not found
Sep 18 18:39:36.965648 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.965579    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s5hn\" (UniqueName: \"kubernetes.io/projected/bf18e0b9-6a23-4338-893f-e8d7ca463c1e-kube-api-access-6s5hn\") pod \"ingress-canary-42z88\" (UID: \"bf18e0b9-6a23-4338-893f-e8d7ca463c1e\") " pod="openshift-ingress-canary/ingress-canary-42z88"
Sep 18 18:39:36.965648 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.965606    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls podName:92f09b3c-b43d-4de1-b12f-b1a1ae026f41 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:37.465589299 +0000 UTC m=+36.902168119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls") pod "image-registry-7bdbd487b5-zx747" (UID: "92f09b3c-b43d-4de1-b12f-b1a1ae026f41") : secret "image-registry-tls" not found
Sep 18 18:39:36.965648 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.965640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-image-registry-private-configuration\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.965917 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.965667    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-trusted-ca\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.965917 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.965757    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6455458b-83a0-4f35-857a-f11c3866299e-tmp-dir\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:36.965917 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.965803    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-installation-pull-secrets\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.965917 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.965841    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-ca-trust-extracted\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.965917 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.965867    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-certificates\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.966295 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.965937    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:36.966295 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.965963    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86xlg\" (UniqueName: \"kubernetes.io/projected/6455458b-83a0-4f35-857a-f11c3866299e-kube-api-access-86xlg\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:36.966295 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.966000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-bound-sa-token\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.966295 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.966245    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6455458b-83a0-4f35-857a-f11c3866299e-config-volume\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:36.966295 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.966166    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-ca-trust-extracted\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.966546 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.966110    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:39:36.966546 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:36.966369    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls podName:6455458b-83a0-4f35-857a-f11c3866299e nodeName:}" failed. No retries permitted until 2026-09-18 18:39:37.466349372 +0000 UTC m=+36.902928248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls") pod "dns-default-zw2cl" (UID: "6455458b-83a0-4f35-857a-f11c3866299e") : secret "dns-default-metrics-tls" not found
Sep 18 18:39:36.966640 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.966567    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-certificates\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.966678 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.966657    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-trusted-ca\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.966850 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.966829    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6455458b-83a0-4f35-857a-f11c3866299e-config-volume\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:36.969588 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.969567    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-installation-pull-secrets\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.969675 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.969589    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf18e0b9-6a23-4338-893f-e8d7ca463c1e-cert\") pod \"ingress-canary-42z88\" (UID: \"bf18e0b9-6a23-4338-893f-e8d7ca463c1e\") " pod="openshift-ingress-canary/ingress-canary-42z88"
Sep 18 18:39:36.969675 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.969629    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-image-registry-private-configuration\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.973795 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.973770    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r8gv\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-kube-api-access-9r8gv\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:36.973940 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.973852    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86xlg\" (UniqueName: \"kubernetes.io/projected/6455458b-83a0-4f35-857a-f11c3866299e-kube-api-access-86xlg\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:36.974163 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.974143    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s5hn\" (UniqueName: \"kubernetes.io/projected/bf18e0b9-6a23-4338-893f-e8d7ca463c1e-kube-api-access-6s5hn\") pod \"ingress-canary-42z88\" (UID: \"bf18e0b9-6a23-4338-893f-e8d7ca463c1e\") " pod="openshift-ingress-canary/ingress-canary-42z88"
Sep 18 18:39:36.974697 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:36.974680    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-bound-sa-token\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:37.054858 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:37.054829    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-42z88"
Sep 18 18:39:37.367403 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:37.367371    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-42z88"]
Sep 18 18:39:37.372644 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:37.372614    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf18e0b9_6a23_4338_893f_e8d7ca463c1e.slice/crio-2136db03cf513d3e780ef0ab2c77bc13cab9b3c3f2d5e68509c3b4f56116f608 WatchSource:0}: Error finding container 2136db03cf513d3e780ef0ab2c77bc13cab9b3c3f2d5e68509c3b4f56116f608: Status 404 returned error can't find the container with id 2136db03cf513d3e780ef0ab2c77bc13cab9b3c3f2d5e68509c3b4f56116f608
Sep 18 18:39:37.470351 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:37.470272    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:37.470351 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:37.470348    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:37.470635 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:37.470447    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 18:39:37.470635 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:37.470460    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:39:37.470635 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:37.470471    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdbd487b5-zx747: secret "image-registry-tls" not found
Sep 18 18:39:37.470635 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:37.470523    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls podName:6455458b-83a0-4f35-857a-f11c3866299e nodeName:}" failed. No retries permitted until 2026-09-18 18:39:38.470509418 +0000 UTC m=+37.907088242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls") pod "dns-default-zw2cl" (UID: "6455458b-83a0-4f35-857a-f11c3866299e") : secret "dns-default-metrics-tls" not found
Sep 18 18:39:37.470635 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:37.470537    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls podName:92f09b3c-b43d-4de1-b12f-b1a1ae026f41 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:38.470531104 +0000 UTC m=+37.907109924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls") pod "image-registry-7bdbd487b5-zx747" (UID: "92f09b3c-b43d-4de1-b12f-b1a1ae026f41") : secret "image-registry-tls" not found
Sep 18 18:39:37.740654 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:37.740568    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-42z88" event={"ID":"bf18e0b9-6a23-4338-893f-e8d7ca463c1e","Type":"ContainerStarted","Data":"2136db03cf513d3e780ef0ab2c77bc13cab9b3c3f2d5e68509c3b4f56116f608"}
Sep 18 18:39:38.481591 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:38.481522    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:38.482346 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:38.481683    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:38.482346 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:38.481702    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:39:38.482346 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:38.481782    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls podName:6455458b-83a0-4f35-857a-f11c3866299e nodeName:}" failed. No retries permitted until 2026-09-18 18:39:40.48175867 +0000 UTC m=+39.918337513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls") pod "dns-default-zw2cl" (UID: "6455458b-83a0-4f35-857a-f11c3866299e") : secret "dns-default-metrics-tls" not found
Sep 18 18:39:38.482346 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:38.481804    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 18:39:38.482346 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:38.481818    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdbd487b5-zx747: secret "image-registry-tls" not found
Sep 18 18:39:38.482346 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:38.481882    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls podName:92f09b3c-b43d-4de1-b12f-b1a1ae026f41 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:40.481856133 +0000 UTC m=+39.918434975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls") pod "image-registry-7bdbd487b5-zx747" (UID: "92f09b3c-b43d-4de1-b12f-b1a1ae026f41") : secret "image-registry-tls" not found
Sep 18 18:39:38.571046 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:38.569842    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:39:38.571046 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:38.569872    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:39:38.571046 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:38.569842    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:39:38.572407 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:38.572384    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 18:39:38.573263 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:38.573244    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 18:39:38.573849 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:38.573503    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5mb7g\""
Sep 18 18:39:38.573849 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:38.573635    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 18:39:38.573849 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:38.573709    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 18:39:38.573849 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:38.573801    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kbgtj\""
Sep 18 18:39:40.500034 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:40.497468    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:40.500034 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:40.497546    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 18:39:40.500034 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:40.497570    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdbd487b5-zx747: secret "image-registry-tls" not found
Sep 18 18:39:40.500034 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:40.497604    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:40.500034 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:40.497641    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls podName:92f09b3c-b43d-4de1-b12f-b1a1ae026f41 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:44.49761296 +0000 UTC m=+43.934191780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls") pod "image-registry-7bdbd487b5-zx747" (UID: "92f09b3c-b43d-4de1-b12f-b1a1ae026f41") : secret "image-registry-tls" not found
Sep 18 18:39:40.500034 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:40.497713    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:39:40.500034 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:40.497773    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls podName:6455458b-83a0-4f35-857a-f11c3866299e nodeName:}" failed. No retries permitted until 2026-09-18 18:39:44.497749451 +0000 UTC m=+43.934328285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls") pod "dns-default-zw2cl" (UID: "6455458b-83a0-4f35-857a-f11c3866299e") : secret "dns-default-metrics-tls" not found
Sep 18 18:39:41.753366 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:41.753332    2558 generic.go:356] "Generic (PLEG): container finished" podID="60babd6b-456a-43cb-81c8-0967234b4c7a" containerID="c6e3605c7c6c077f32d53fb24daf55c84d6fdb682736614e687d4e2b5eac5df1" exitCode=0
Sep 18 18:39:41.753793 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:41.753403    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thdgp" event={"ID":"60babd6b-456a-43cb-81c8-0967234b4c7a","Type":"ContainerDied","Data":"c6e3605c7c6c077f32d53fb24daf55c84d6fdb682736614e687d4e2b5eac5df1"}
Sep 18 18:39:41.756776 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:41.756687    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-42z88" event={"ID":"bf18e0b9-6a23-4338-893f-e8d7ca463c1e","Type":"ContainerStarted","Data":"6b960355631af2a9a0b3b4445404cc8c415570b62c3f0ba7ac161e6046a396a8"}
Sep 18 18:39:41.789427 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:41.789379    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-42z88" podStartSLOduration=2.344694771 podStartE2EDuration="5.789362993s" totalImagesPullingTime="3.444668222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:41.788856088 +0000 UTC m=+41.225434931" watchObservedRunningTime="2026-09-18 18:39:41.789362993 +0000 UTC m=+41.225941835"
Sep 18 18:39:42.764695 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:42.764660    2558 generic.go:356] "Generic (PLEG): container finished" podID="60babd6b-456a-43cb-81c8-0967234b4c7a" containerID="67459719885cf51efa8985827e369deba47a37169c59299ab559008ffcfde987" exitCode=0
Sep 18 18:39:42.765104 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:42.764737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thdgp" event={"ID":"60babd6b-456a-43cb-81c8-0967234b4c7a","Type":"ContainerDied","Data":"67459719885cf51efa8985827e369deba47a37169c59299ab559008ffcfde987"}
Sep 18 18:39:43.065299 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.065228    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cltgd"]
Sep 18 18:39:43.068769 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.068754    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cltgd"
Sep 18 18:39:43.070002 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.069977    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"]
Sep 18 18:39:43.071602 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.071587    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5b98q\""
Sep 18 18:39:43.071943 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.071926    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 18:39:43.071998 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.071982    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 18:39:43.074162 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.074145    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"
Sep 18 18:39:43.075262 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.075244    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-whw2g"]
Sep 18 18:39:43.076425 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.076403    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 18:39:43.076511 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.076464    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-n2h7g\""
Sep 18 18:39:43.076728 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.076713    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 18 18:39:43.079364 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.079343    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6bd44bd444-kdgsw"]
Sep 18 18:39:43.079461 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.079445    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:43.082051 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.082034    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 18:39:43.082264 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.082245    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 18:39:43.082328 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.082289    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5bsl2\""
Sep 18 18:39:43.082495 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.082482    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 18:39:43.082709 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.082691    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 18:39:43.083356 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.083337    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"]
Sep 18 18:39:43.083483 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.083466    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.086661 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.086641    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 18:39:43.086875 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.086855    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 18:39:43.086978 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.086890    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 18:39:43.087043 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.086991    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 18:39:43.087098 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.087088    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 18:39:43.087153 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.087105    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 18:39:43.087304 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.087289    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-k7f25\""
Sep 18 18:39:43.087378 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.087330    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cltgd"]
Sep 18 18:39:43.087378 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.087363    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"]
Sep 18 18:39:43.087512 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.087441    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:39:43.089587 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.089565    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 18:39:43.090607 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.090585    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mq7c7\""
Sep 18 18:39:43.090689 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.090625    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 18:39:43.090689 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.090698    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 18:39:43.090806 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.090776    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 18:39:43.091869 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.091853    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 18:39:43.093834 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.093806    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-whw2g"]
Sep 18 18:39:43.094569 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.094544    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"]
Sep 18 18:39:43.102501 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.102482    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6bd44bd444-kdgsw"]
Sep 18 18:39:43.170339 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.170308    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj"]
Sep 18 18:39:43.174392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.174371    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj"
Sep 18 18:39:43.174546 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.174394    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr"]
Sep 18 18:39:43.177486 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.177471    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 18:39:43.177967 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.177951    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 18:39:43.178139 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.178123    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-nd8ht"]
Sep 18 18:39:43.178230 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.178144    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 18:39:43.178284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.178240    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr"
Sep 18 18:39:43.178349 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.178336    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9b8kb\""
Sep 18 18:39:43.178442 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.178421    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 18:39:43.180774 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.180754    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 18:39:43.181266 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.181250    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 18:39:43.181651 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.181633    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-q64fh"]
Sep 18 18:39:43.181749 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.181735    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nd8ht"
Sep 18 18:39:43.182069 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.182048    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 18:39:43.182148 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.182111    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 18:39:43.182228 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.182205    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2t44d\""
Sep 18 18:39:43.184959 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.184937    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qqllv\""
Sep 18 18:39:43.185093 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.185080    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.188097 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.188077    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 18:39:43.188215 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.188124    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 18:39:43.189962 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.189610    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 18:39:43.189962 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.189809    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj"]
Sep 18 18:39:43.189962 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.189948    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qxhn8\""
Sep 18 18:39:43.190166 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.189990    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 18:39:43.190658 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.190639    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr"]
Sep 18 18:39:43.191365 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.191345    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-nd8ht"]
Sep 18 18:39:43.194034 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.193998    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-q64fh"]
Sep 18 18:39:43.195381 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.195365    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 18:39:43.216694 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.216676    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxwf4\" (UniqueName: \"kubernetes.io/projected/9b2a9f30-7931-4049-8d50-64203df5c3fd-kube-api-access-kxwf4\") pod \"cluster-monitoring-operator-847b8bd8d7-p27wv\" (UID: \"9b2a9f30-7931-4049-8d50-64203df5c3fd\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:39:43.216785 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.216712    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjv89\" (UniqueName: \"kubernetes.io/projected/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-kube-api-access-qjv89\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.216785 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.216736    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c3a44de-98c4-42e8-9ed2-7587293418ca-config\") pod \"console-operator-fd4bb46c-whw2g\" (UID: \"8c3a44de-98c4-42e8-9ed2-7587293418ca\") " pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:43.216785 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.216777    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-stats-auth\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.216884 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.216809    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.216884 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.216854    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4lcv7\" (UID: \"f649578f-b8d4-4207-9dcc-69ce583fc6f6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"
Sep 18 18:39:43.216884 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.216876    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvc98\" (UniqueName: \"kubernetes.io/projected/8c3a44de-98c4-42e8-9ed2-7587293418ca-kube-api-access-zvc98\") pod \"console-operator-fd4bb46c-whw2g\" (UID: \"8c3a44de-98c4-42e8-9ed2-7587293418ca\") " pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:43.216970 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.216911    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.216970 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.216935    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c3a44de-98c4-42e8-9ed2-7587293418ca-serving-cert\") pod \"console-operator-fd4bb46c-whw2g\" (UID: \"8c3a44de-98c4-42e8-9ed2-7587293418ca\") " pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:43.216970 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.216967    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f649578f-b8d4-4207-9dcc-69ce583fc6f6-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-4lcv7\" (UID: \"f649578f-b8d4-4207-9dcc-69ce583fc6f6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"
Sep 18 18:39:43.217053 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.216982    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-p27wv\" (UID: \"9b2a9f30-7931-4049-8d50-64203df5c3fd\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:39:43.217053 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.217000    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-default-certificate\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.217053 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.217017    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c3a44de-98c4-42e8-9ed2-7587293418ca-trusted-ca\") pod \"console-operator-fd4bb46c-whw2g\" (UID: \"8c3a44de-98c4-42e8-9ed2-7587293418ca\") " pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:43.217053 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.217037    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9ztm\" (UniqueName: \"kubernetes.io/projected/f0bb329d-2119-4411-bdbd-8c01fdcf493e-kube-api-access-b9ztm\") pod \"volume-data-source-validator-5c4db8bdf9-cltgd\" (UID: \"f0bb329d-2119-4411-bdbd-8c01fdcf493e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cltgd"
Sep 18 18:39:43.217165 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.217062    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9b2a9f30-7931-4049-8d50-64203df5c3fd-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-p27wv\" (UID: \"9b2a9f30-7931-4049-8d50-64203df5c3fd\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:39:43.269409 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.269386    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"]
Sep 18 18:39:43.273116 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.273098    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"
Sep 18 18:39:43.275399 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.275376    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 18:39:43.275505 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.275400    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wvdss\""
Sep 18 18:39:43.275505 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.275416    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 18:39:43.275505 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.275420    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 18:39:43.285399 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.285380    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"]
Sep 18 18:39:43.317464 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.317403    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.317464 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.317432    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c3a44de-98c4-42e8-9ed2-7587293418ca-serving-cert\") pod \"console-operator-fd4bb46c-whw2g\" (UID: \"8c3a44de-98c4-42e8-9ed2-7587293418ca\") " pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:43.317464 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.317452    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/63302165-e325-4efd-88d8-dda368f6066e-serving-cert\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.317629 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.317552    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 18:39:43.317629 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.317597    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f649578f-b8d4-4207-9dcc-69ce583fc6f6-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-4lcv7\" (UID: \"f649578f-b8d4-4207-9dcc-69ce583fc6f6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"
Sep 18 18:39:43.317694 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.317638    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-p27wv\" (UID: \"9b2a9f30-7931-4049-8d50-64203df5c3fd\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:39:43.317694 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.317655    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs podName:7117c313-6fb4-48b0-8f3e-16e2c9c8d551 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:43.817635507 +0000 UTC m=+43.254214340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs") pod "router-default-6bd44bd444-kdgsw" (UID: "7117c313-6fb4-48b0-8f3e-16e2c9c8d551") : secret "router-metrics-certs-default" not found
Sep 18 18:39:43.317694 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.317688    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-default-certificate\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.317818 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.317707    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 18:39:43.317818 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.317717    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c3a44de-98c4-42e8-9ed2-7587293418ca-trusted-ca\") pod \"console-operator-fd4bb46c-whw2g\" (UID: \"8c3a44de-98c4-42e8-9ed2-7587293418ca\") " pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:43.317818 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.317749    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls podName:9b2a9f30-7931-4049-8d50-64203df5c3fd nodeName:}" failed. No retries permitted until 2026-09-18 18:39:43.817738102 +0000 UTC m=+43.254316925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-p27wv" (UID: "9b2a9f30-7931-4049-8d50-64203df5c3fd") : secret "cluster-monitoring-operator-tls" not found
Sep 18 18:39:43.317818 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.317765    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63302165-e325-4efd-88d8-dda368f6066e-tmp\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.317818 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.317798    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9ztm\" (UniqueName: \"kubernetes.io/projected/f0bb329d-2119-4411-bdbd-8c01fdcf493e-kube-api-access-b9ztm\") pod \"volume-data-source-validator-5c4db8bdf9-cltgd\" (UID: \"f0bb329d-2119-4411-bdbd-8c01fdcf493e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cltgd"
Sep 18 18:39:43.318037 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.317846    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9b2a9f30-7931-4049-8d50-64203df5c3fd-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-p27wv\" (UID: \"9b2a9f30-7931-4049-8d50-64203df5c3fd\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:39:43.318037 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.317935    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z857h\" (UniqueName: \"kubernetes.io/projected/3ff38d46-064b-4f8a-b34b-66a6f125e317-kube-api-access-z857h\") pod \"network-check-source-7cbc45fddc-nd8ht\" (UID: \"3ff38d46-064b-4f8a-b34b-66a6f125e317\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nd8ht"
Sep 18 18:39:43.318037 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxwf4\" (UniqueName: \"kubernetes.io/projected/9b2a9f30-7931-4049-8d50-64203df5c3fd-kube-api-access-kxwf4\") pod \"cluster-monitoring-operator-847b8bd8d7-p27wv\" (UID: \"9b2a9f30-7931-4049-8d50-64203df5c3fd\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:39:43.318181 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318042    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63302165-e325-4efd-88d8-dda368f6066e-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.318181 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318131    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cccc2e3c-0d90-42aa-a136-069631e71567-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-84wlr\" (UID: \"cccc2e3c-0d90-42aa-a136-069631e71567\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr"
Sep 18 18:39:43.318311 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318177    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cccc2e3c-0d90-42aa-a136-069631e71567-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-84wlr\" (UID: \"cccc2e3c-0d90-42aa-a136-069631e71567\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr"
Sep 18 18:39:43.318311 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318243    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjv89\" (UniqueName: \"kubernetes.io/projected/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-kube-api-access-qjv89\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.318311 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318289    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8b41f87c-1c13-44ea-ae32-8ff22711e45e-serving-cert\") pod \"service-ca-operator-57cdc4b768-n7tgj\" (UID: \"8b41f87c-1c13-44ea-ae32-8ff22711e45e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj"
Sep 18 18:39:43.318468 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318332    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c3a44de-98c4-42e8-9ed2-7587293418ca-config\") pod \"console-operator-fd4bb46c-whw2g\" (UID: \"8c3a44de-98c4-42e8-9ed2-7587293418ca\") " pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:43.318468 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318349    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f649578f-b8d4-4207-9dcc-69ce583fc6f6-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-4lcv7\" (UID: \"f649578f-b8d4-4207-9dcc-69ce583fc6f6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"
Sep 18 18:39:43.318468 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318373    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk6rp\" (UniqueName: \"kubernetes.io/projected/63302165-e325-4efd-88d8-dda368f6066e-kube-api-access-zk6rp\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.318468 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318435    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-stats-auth\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.318638 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318571    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8b41f87c-1c13-44ea-ae32-8ff22711e45e-config\") pod \"service-ca-operator-57cdc4b768-n7tgj\" (UID: \"8b41f87c-1c13-44ea-ae32-8ff22711e45e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj"
Sep 18 18:39:43.318638 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318609    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c3a44de-98c4-42e8-9ed2-7587293418ca-trusted-ca\") pod \"console-operator-fd4bb46c-whw2g\" (UID: \"8c3a44de-98c4-42e8-9ed2-7587293418ca\") " pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:43.318726 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318632    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfrqb\" (UniqueName: \"kubernetes.io/projected/cccc2e3c-0d90-42aa-a136-069631e71567-kube-api-access-lfrqb\") pod \"kube-storage-version-migrator-operator-55889f7b47-84wlr\" (UID: \"cccc2e3c-0d90-42aa-a136-069631e71567\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr"
Sep 18 18:39:43.318726 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318683    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.318726 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318715    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9b2a9f30-7931-4049-8d50-64203df5c3fd-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-p27wv\" (UID: \"9b2a9f30-7931-4049-8d50-64203df5c3fd\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:39:43.318875 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318716    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqv4n\" (UniqueName: \"kubernetes.io/projected/8b41f87c-1c13-44ea-ae32-8ff22711e45e-kube-api-access-vqv4n\") pod \"service-ca-operator-57cdc4b768-n7tgj\" (UID: \"8b41f87c-1c13-44ea-ae32-8ff22711e45e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj"
Sep 18 18:39:43.318875 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.318817    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle podName:7117c313-6fb4-48b0-8f3e-16e2c9c8d551 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:43.818799343 +0000 UTC m=+43.255378164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle") pod "router-default-6bd44bd444-kdgsw" (UID: "7117c313-6fb4-48b0-8f3e-16e2c9c8d551") : configmap references non-existent config key: service-ca.crt
Sep 18 18:39:43.318875 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318867    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/63302165-e325-4efd-88d8-dda368f6066e-snapshots\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.319030 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318922    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63302165-e325-4efd-88d8-dda368f6066e-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.319030 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.318977    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4lcv7\" (UID: \"f649578f-b8d4-4207-9dcc-69ce583fc6f6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"
Sep 18 18:39:43.319030 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.319015    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvc98\" (UniqueName: \"kubernetes.io/projected/8c3a44de-98c4-42e8-9ed2-7587293418ca-kube-api-access-zvc98\") pod \"console-operator-fd4bb46c-whw2g\" (UID: \"8c3a44de-98c4-42e8-9ed2-7587293418ca\") " pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:43.319168 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.319026    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c3a44de-98c4-42e8-9ed2-7587293418ca-config\") pod \"console-operator-fd4bb46c-whw2g\" (UID: \"8c3a44de-98c4-42e8-9ed2-7587293418ca\") " pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:43.319168 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.319103    2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 18:39:43.319292 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.319217    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert podName:f649578f-b8d4-4207-9dcc-69ce583fc6f6 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:43.819201276 +0000 UTC m=+43.255780112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4lcv7" (UID: "f649578f-b8d4-4207-9dcc-69ce583fc6f6") : secret "networking-console-plugin-cert" not found
Sep 18 18:39:43.320481 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.320453    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8c3a44de-98c4-42e8-9ed2-7587293418ca-serving-cert\") pod \"console-operator-fd4bb46c-whw2g\" (UID: \"8c3a44de-98c4-42e8-9ed2-7587293418ca\") " pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:43.321103 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.321076    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-stats-auth\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.321103 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.321083    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-default-certificate\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.326929 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.326901    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9ztm\" (UniqueName: \"kubernetes.io/projected/f0bb329d-2119-4411-bdbd-8c01fdcf493e-kube-api-access-b9ztm\") pod \"volume-data-source-validator-5c4db8bdf9-cltgd\" (UID: \"f0bb329d-2119-4411-bdbd-8c01fdcf493e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cltgd"
Sep 18 18:39:43.327251 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.327224    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjv89\" (UniqueName: \"kubernetes.io/projected/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-kube-api-access-qjv89\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.327328 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.327274    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxwf4\" (UniqueName: \"kubernetes.io/projected/9b2a9f30-7931-4049-8d50-64203df5c3fd-kube-api-access-kxwf4\") pod \"cluster-monitoring-operator-847b8bd8d7-p27wv\" (UID: \"9b2a9f30-7931-4049-8d50-64203df5c3fd\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:39:43.327379 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.327364    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvc98\" (UniqueName: \"kubernetes.io/projected/8c3a44de-98c4-42e8-9ed2-7587293418ca-kube-api-access-zvc98\") pod \"console-operator-fd4bb46c-whw2g\" (UID: \"8c3a44de-98c4-42e8-9ed2-7587293418ca\") " pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:43.379171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.379146    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cltgd"
Sep 18 18:39:43.397294 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.397264    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:43.421521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420257    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8b41f87c-1c13-44ea-ae32-8ff22711e45e-config\") pod \"service-ca-operator-57cdc4b768-n7tgj\" (UID: \"8b41f87c-1c13-44ea-ae32-8ff22711e45e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj"
Sep 18 18:39:43.421521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420296    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfrqb\" (UniqueName: \"kubernetes.io/projected/cccc2e3c-0d90-42aa-a136-069631e71567-kube-api-access-lfrqb\") pod \"kube-storage-version-migrator-operator-55889f7b47-84wlr\" (UID: \"cccc2e3c-0d90-42aa-a136-069631e71567\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr"
Sep 18 18:39:43.421521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420328    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqv4n\" (UniqueName: \"kubernetes.io/projected/8b41f87c-1c13-44ea-ae32-8ff22711e45e-kube-api-access-vqv4n\") pod \"service-ca-operator-57cdc4b768-n7tgj\" (UID: \"8b41f87c-1c13-44ea-ae32-8ff22711e45e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj"
Sep 18 18:39:43.421521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420343    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/63302165-e325-4efd-88d8-dda368f6066e-snapshots\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.421521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420365    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63302165-e325-4efd-88d8-dda368f6066e-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.421521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420409    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zqkl5\" (UID: \"51902e02-aefc-4f2a-bab5-b65c1f981852\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"
Sep 18 18:39:43.421521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420430    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/63302165-e325-4efd-88d8-dda368f6066e-serving-cert\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.421521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420481    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63302165-e325-4efd-88d8-dda368f6066e-tmp\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.421521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420506    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z857h\" (UniqueName: \"kubernetes.io/projected/3ff38d46-064b-4f8a-b34b-66a6f125e317-kube-api-access-z857h\") pod \"network-check-source-7cbc45fddc-nd8ht\" (UID: \"3ff38d46-064b-4f8a-b34b-66a6f125e317\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nd8ht"
Sep 18 18:39:43.421521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420538    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63302165-e325-4efd-88d8-dda368f6066e-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.421521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420564    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cccc2e3c-0d90-42aa-a136-069631e71567-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-84wlr\" (UID: \"cccc2e3c-0d90-42aa-a136-069631e71567\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr"
Sep 18 18:39:43.421521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420583    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cccc2e3c-0d90-42aa-a136-069631e71567-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-84wlr\" (UID: \"cccc2e3c-0d90-42aa-a136-069631e71567\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr"
Sep 18 18:39:43.421521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420606    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8pmq\" (UniqueName: \"kubernetes.io/projected/51902e02-aefc-4f2a-bab5-b65c1f981852-kube-api-access-h8pmq\") pod \"cluster-samples-operator-685d7d68b5-zqkl5\" (UID: \"51902e02-aefc-4f2a-bab5-b65c1f981852\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"
Sep 18 18:39:43.421521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420628    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8b41f87c-1c13-44ea-ae32-8ff22711e45e-serving-cert\") pod \"service-ca-operator-57cdc4b768-n7tgj\" (UID: \"8b41f87c-1c13-44ea-ae32-8ff22711e45e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj"
Sep 18 18:39:43.421521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420656    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk6rp\" (UniqueName: \"kubernetes.io/projected/63302165-e325-4efd-88d8-dda368f6066e-kube-api-access-zk6rp\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.422466 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.420678    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8b41f87c-1c13-44ea-ae32-8ff22711e45e-config\") pod \"service-ca-operator-57cdc4b768-n7tgj\" (UID: \"8b41f87c-1c13-44ea-ae32-8ff22711e45e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj"
Sep 18 18:39:43.422466 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.421364    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/63302165-e325-4efd-88d8-dda368f6066e-snapshots\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.422466 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.421987    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63302165-e325-4efd-88d8-dda368f6066e-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.422466 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.422263    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63302165-e325-4efd-88d8-dda368f6066e-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.422466 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.422377    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63302165-e325-4efd-88d8-dda368f6066e-tmp\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.422695 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.422539    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cccc2e3c-0d90-42aa-a136-069631e71567-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-84wlr\" (UID: \"cccc2e3c-0d90-42aa-a136-069631e71567\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr"
Sep 18 18:39:43.423773 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.423755    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8b41f87c-1c13-44ea-ae32-8ff22711e45e-serving-cert\") pod \"service-ca-operator-57cdc4b768-n7tgj\" (UID: \"8b41f87c-1c13-44ea-ae32-8ff22711e45e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj"
Sep 18 18:39:43.424142 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.424127    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/63302165-e325-4efd-88d8-dda368f6066e-serving-cert\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.424371 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.424348    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cccc2e3c-0d90-42aa-a136-069631e71567-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-84wlr\" (UID: \"cccc2e3c-0d90-42aa-a136-069631e71567\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr"
Sep 18 18:39:43.431769 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.428915    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk6rp\" (UniqueName: \"kubernetes.io/projected/63302165-e325-4efd-88d8-dda368f6066e-kube-api-access-zk6rp\") pod \"insights-operator-6f9c644dd9-q64fh\" (UID: \"63302165-e325-4efd-88d8-dda368f6066e\") " pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.431769 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.431408    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfrqb\" (UniqueName: \"kubernetes.io/projected/cccc2e3c-0d90-42aa-a136-069631e71567-kube-api-access-lfrqb\") pod \"kube-storage-version-migrator-operator-55889f7b47-84wlr\" (UID: \"cccc2e3c-0d90-42aa-a136-069631e71567\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr"
Sep 18 18:39:43.432335 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.431806    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqv4n\" (UniqueName: \"kubernetes.io/projected/8b41f87c-1c13-44ea-ae32-8ff22711e45e-kube-api-access-vqv4n\") pod \"service-ca-operator-57cdc4b768-n7tgj\" (UID: \"8b41f87c-1c13-44ea-ae32-8ff22711e45e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj"
Sep 18 18:39:43.433806 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.433779    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z857h\" (UniqueName: \"kubernetes.io/projected/3ff38d46-064b-4f8a-b34b-66a6f125e317-kube-api-access-z857h\") pod \"network-check-source-7cbc45fddc-nd8ht\" (UID: \"3ff38d46-064b-4f8a-b34b-66a6f125e317\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nd8ht"
Sep 18 18:39:43.484546 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.484519    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj"
Sep 18 18:39:43.492175 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.492158    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr"
Sep 18 18:39:43.499414 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.499394    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nd8ht"
Sep 18 18:39:43.504530 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.504504    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-q64fh"
Sep 18 18:39:43.522222 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.522144    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zqkl5\" (UID: \"51902e02-aefc-4f2a-bab5-b65c1f981852\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"
Sep 18 18:39:43.522339 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.522219    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 18:39:43.522339 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.522297    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls podName:51902e02-aefc-4f2a-bab5-b65c1f981852 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:44.022277534 +0000 UTC m=+43.458856371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-zqkl5" (UID: "51902e02-aefc-4f2a-bab5-b65c1f981852") : secret "samples-operator-tls" not found
Sep 18 18:39:43.522437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.522385    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8pmq\" (UniqueName: \"kubernetes.io/projected/51902e02-aefc-4f2a-bab5-b65c1f981852-kube-api-access-h8pmq\") pod \"cluster-samples-operator-685d7d68b5-zqkl5\" (UID: \"51902e02-aefc-4f2a-bab5-b65c1f981852\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"
Sep 18 18:39:43.534856 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.534784    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8pmq\" (UniqueName: \"kubernetes.io/projected/51902e02-aefc-4f2a-bab5-b65c1f981852-kube-api-access-h8pmq\") pod \"cluster-samples-operator-685d7d68b5-zqkl5\" (UID: \"51902e02-aefc-4f2a-bab5-b65c1f981852\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"
Sep 18 18:39:43.650610 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.650580    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cltgd"]
Sep 18 18:39:43.654037 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:43.654010    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bb329d_2119_4411_bdbd_8c01fdcf493e.slice/crio-e77e0ec886d66779c9407eea47f1900c26e44b8a816a52c56fcb8ba196fbf50a WatchSource:0}: Error finding container e77e0ec886d66779c9407eea47f1900c26e44b8a816a52c56fcb8ba196fbf50a: Status 404 returned error can't find the container with id e77e0ec886d66779c9407eea47f1900c26e44b8a816a52c56fcb8ba196fbf50a
Sep 18 18:39:43.673344 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.673322    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-whw2g"]
Sep 18 18:39:43.675722 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:43.675698    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3a44de_98c4_42e8_9ed2_7587293418ca.slice/crio-00bbd56adc5ff389b3f608f7a38849111bfc4562d0240ac16b9cdace7275248e WatchSource:0}: Error finding container 00bbd56adc5ff389b3f608f7a38849111bfc4562d0240ac16b9cdace7275248e: Status 404 returned error can't find the container with id 00bbd56adc5ff389b3f608f7a38849111bfc4562d0240ac16b9cdace7275248e
Sep 18 18:39:43.761366 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.761335    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj"]
Sep 18 18:39:43.764269 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:43.764244    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b41f87c_1c13_44ea_ae32_8ff22711e45e.slice/crio-e31afada7af383868126c9ae4fae039bd7db957cda87ac5c510f48914f1708fe WatchSource:0}: Error finding container e31afada7af383868126c9ae4fae039bd7db957cda87ac5c510f48914f1708fe: Status 404 returned error can't find the container with id e31afada7af383868126c9ae4fae039bd7db957cda87ac5c510f48914f1708fe
Sep 18 18:39:43.770412 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.770378    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-thdgp" event={"ID":"60babd6b-456a-43cb-81c8-0967234b4c7a","Type":"ContainerStarted","Data":"5700129e617197717f3392ba725eeab40eeaee1f062489fd4c02640567347ca1"}
Sep 18 18:39:43.771625 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.771599    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj" event={"ID":"8b41f87c-1c13-44ea-ae32-8ff22711e45e","Type":"ContainerStarted","Data":"e31afada7af383868126c9ae4fae039bd7db957cda87ac5c510f48914f1708fe"}
Sep 18 18:39:43.772664 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.772642    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-whw2g" event={"ID":"8c3a44de-98c4-42e8-9ed2-7587293418ca","Type":"ContainerStarted","Data":"00bbd56adc5ff389b3f608f7a38849111bfc4562d0240ac16b9cdace7275248e"}
Sep 18 18:39:43.773686 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.773664    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cltgd" event={"ID":"f0bb329d-2119-4411-bdbd-8c01fdcf493e","Type":"ContainerStarted","Data":"e77e0ec886d66779c9407eea47f1900c26e44b8a816a52c56fcb8ba196fbf50a"}
Sep 18 18:39:43.791674 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.791637    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-thdgp" podStartSLOduration=8.823127741 podStartE2EDuration="41.791622183s" totalImagesPullingTime="32.968494442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:39:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:43.791380155 +0000 UTC m=+43.227959013" watchObservedRunningTime="2026-09-18 18:39:43.791622183 +0000 UTC m=+43.228201030"
Sep 18 18:39:43.799492 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.799471    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-nd8ht"]
Sep 18 18:39:43.802351 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.802329    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr"]
Sep 18 18:39:43.802736 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:43.802715    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff38d46_064b_4f8a_b34b_66a6f125e317.slice/crio-c952a493d711913df95258a1c764535e25cd3253dda94d405e51ea4a4880ec80 WatchSource:0}: Error finding container c952a493d711913df95258a1c764535e25cd3253dda94d405e51ea4a4880ec80: Status 404 returned error can't find the container with id c952a493d711913df95258a1c764535e25cd3253dda94d405e51ea4a4880ec80
Sep 18 18:39:43.804755 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:43.804737    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccc2e3c_0d90_42aa_a136_069631e71567.slice/crio-02ed81a9df6d008e25855bddef3dee37d11c8c74bf90e8f7fbdc6a9e4223a954 WatchSource:0}: Error finding container 02ed81a9df6d008e25855bddef3dee37d11c8c74bf90e8f7fbdc6a9e4223a954: Status 404 returned error can't find the container with id 02ed81a9df6d008e25855bddef3dee37d11c8c74bf90e8f7fbdc6a9e4223a954
Sep 18 18:39:43.817573 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.817542    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-q64fh"]
Sep 18 18:39:43.820808 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:43.820786    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63302165_e325_4efd_88d8_dda368f6066e.slice/crio-7821c1bbca305c92b3b1dfa6a7d38bc9e6f454f53cb68aa6396ba6b11c524fdd WatchSource:0}: Error finding container 7821c1bbca305c92b3b1dfa6a7d38bc9e6f454f53cb68aa6396ba6b11c524fdd: Status 404 returned error can't find the container with id 7821c1bbca305c92b3b1dfa6a7d38bc9e6f454f53cb68aa6396ba6b11c524fdd
Sep 18 18:39:43.825372 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.825353    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4lcv7\" (UID: \"f649578f-b8d4-4207-9dcc-69ce583fc6f6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"
Sep 18 18:39:43.825435 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.825390    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.825520 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.825503    2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 18:39:43.825581 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.825526    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-p27wv\" (UID: \"9b2a9f30-7931-4049-8d50-64203df5c3fd\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:39:43.825581 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.825511    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 18:39:43.825581 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.825568    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert podName:f649578f-b8d4-4207-9dcc-69ce583fc6f6 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:44.825547279 +0000 UTC m=+44.262126099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4lcv7" (UID: "f649578f-b8d4-4207-9dcc-69ce583fc6f6") : secret "networking-console-plugin-cert" not found
Sep 18 18:39:43.825581 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.825576    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 18:39:43.825714 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.825607    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs podName:7117c313-6fb4-48b0-8f3e-16e2c9c8d551 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:44.825591294 +0000 UTC m=+44.262170115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs") pod "router-default-6bd44bd444-kdgsw" (UID: "7117c313-6fb4-48b0-8f3e-16e2c9c8d551") : secret "router-metrics-certs-default" not found
Sep 18 18:39:43.825714 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:43.825677    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:43.825714 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.825707    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls podName:9b2a9f30-7931-4049-8d50-64203df5c3fd nodeName:}" failed. No retries permitted until 2026-09-18 18:39:44.825688445 +0000 UTC m=+44.262267279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-p27wv" (UID: "9b2a9f30-7931-4049-8d50-64203df5c3fd") : secret "cluster-monitoring-operator-tls" not found
Sep 18 18:39:43.825819 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:43.825760    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle podName:7117c313-6fb4-48b0-8f3e-16e2c9c8d551 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:44.82575093 +0000 UTC m=+44.262329749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle") pod "router-default-6bd44bd444-kdgsw" (UID: "7117c313-6fb4-48b0-8f3e-16e2c9c8d551") : configmap references non-existent config key: service-ca.crt
Sep 18 18:39:44.027544 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:44.027490    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zqkl5\" (UID: \"51902e02-aefc-4f2a-bab5-b65c1f981852\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"
Sep 18 18:39:44.027734 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:44.027648    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 18:39:44.027734 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:44.027732    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls podName:51902e02-aefc-4f2a-bab5-b65c1f981852 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:45.027713058 +0000 UTC m=+44.464291878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-zqkl5" (UID: "51902e02-aefc-4f2a-bab5-b65c1f981852") : secret "samples-operator-tls" not found
Sep 18 18:39:44.532726 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:44.532688    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:44.532927 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:44.532791    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:44.532991 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:44.532949    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:39:44.533041 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:44.533018    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls podName:6455458b-83a0-4f35-857a-f11c3866299e nodeName:}" failed. No retries permitted until 2026-09-18 18:39:52.53299755 +0000 UTC m=+51.969576387 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls") pod "dns-default-zw2cl" (UID: "6455458b-83a0-4f35-857a-f11c3866299e") : secret "dns-default-metrics-tls" not found
Sep 18 18:39:44.533544 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:44.533520    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 18:39:44.533544 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:44.533546    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdbd487b5-zx747: secret "image-registry-tls" not found
Sep 18 18:39:44.533705 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:44.533594    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls podName:92f09b3c-b43d-4de1-b12f-b1a1ae026f41 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:52.533577716 +0000 UTC m=+51.970156558 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls") pod "image-registry-7bdbd487b5-zx747" (UID: "92f09b3c-b43d-4de1-b12f-b1a1ae026f41") : secret "image-registry-tls" not found
Sep 18 18:39:44.781211 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:44.781150    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nd8ht" event={"ID":"3ff38d46-064b-4f8a-b34b-66a6f125e317","Type":"ContainerStarted","Data":"c952a493d711913df95258a1c764535e25cd3253dda94d405e51ea4a4880ec80"}
Sep 18 18:39:44.783084 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:44.783015    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-q64fh" event={"ID":"63302165-e325-4efd-88d8-dda368f6066e","Type":"ContainerStarted","Data":"7821c1bbca305c92b3b1dfa6a7d38bc9e6f454f53cb68aa6396ba6b11c524fdd"}
Sep 18 18:39:44.787621 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:44.787591    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr" event={"ID":"cccc2e3c-0d90-42aa-a136-069631e71567","Type":"ContainerStarted","Data":"02ed81a9df6d008e25855bddef3dee37d11c8c74bf90e8f7fbdc6a9e4223a954"}
Sep 18 18:39:44.835256 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:44.835216    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:44.835418 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:44.835271    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4lcv7\" (UID: \"f649578f-b8d4-4207-9dcc-69ce583fc6f6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"
Sep 18 18:39:44.835418 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:44.835310    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:44.835418 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:44.835374    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-p27wv\" (UID: \"9b2a9f30-7931-4049-8d50-64203df5c3fd\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:39:44.835606 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:44.835562    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 18:39:44.835661 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:44.835629    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls podName:9b2a9f30-7931-4049-8d50-64203df5c3fd nodeName:}" failed. No retries permitted until 2026-09-18 18:39:46.835609668 +0000 UTC m=+46.272188491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-p27wv" (UID: "9b2a9f30-7931-4049-8d50-64203df5c3fd") : secret "cluster-monitoring-operator-tls" not found
Sep 18 18:39:44.836084 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:44.836064    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle podName:7117c313-6fb4-48b0-8f3e-16e2c9c8d551 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:46.836046539 +0000 UTC m=+46.272625382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle") pod "router-default-6bd44bd444-kdgsw" (UID: "7117c313-6fb4-48b0-8f3e-16e2c9c8d551") : configmap references non-existent config key: service-ca.crt
Sep 18 18:39:44.836212 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:44.836138    2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 18:39:44.836212 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:44.836178    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert podName:f649578f-b8d4-4207-9dcc-69ce583fc6f6 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:46.836166959 +0000 UTC m=+46.272745781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4lcv7" (UID: "f649578f-b8d4-4207-9dcc-69ce583fc6f6") : secret "networking-console-plugin-cert" not found
Sep 18 18:39:44.836332 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:44.836247    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 18:39:44.836332 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:44.836281    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs podName:7117c313-6fb4-48b0-8f3e-16e2c9c8d551 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:46.836270284 +0000 UTC m=+46.272849122 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs") pod "router-default-6bd44bd444-kdgsw" (UID: "7117c313-6fb4-48b0-8f3e-16e2c9c8d551") : secret "router-metrics-certs-default" not found
Sep 18 18:39:45.037572 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:45.037454    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zqkl5\" (UID: \"51902e02-aefc-4f2a-bab5-b65c1f981852\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"
Sep 18 18:39:45.037745 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:45.037631    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 18:39:45.037745 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:45.037721    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls podName:51902e02-aefc-4f2a-bab5-b65c1f981852 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:47.037699046 +0000 UTC m=+46.474277876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-zqkl5" (UID: "51902e02-aefc-4f2a-bab5-b65c1f981852") : secret "samples-operator-tls" not found
Sep 18 18:39:45.791765 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:45.791686    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cltgd" event={"ID":"f0bb329d-2119-4411-bdbd-8c01fdcf493e","Type":"ContainerStarted","Data":"95db76d132ac3ba1ffa1c05622c37fc768d9a02689d249e914392f98afad0b01"}
Sep 18 18:39:45.807596 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:45.807550    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-cltgd" podStartSLOduration=1.0915875879999999 podStartE2EDuration="2.807535831s" totalImagesPullingTime="1.715948243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:45.806522886 +0000 UTC m=+45.243101732" watchObservedRunningTime="2026-09-18 18:39:45.807535831 +0000 UTC m=+45.244114672"
Sep 18 18:39:46.856598 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:46.856556    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:46.857016 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:46.856612    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4lcv7\" (UID: \"f649578f-b8d4-4207-9dcc-69ce583fc6f6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"
Sep 18 18:39:46.857016 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:46.856649    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:46.857016 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:46.856703    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-p27wv\" (UID: \"9b2a9f30-7931-4049-8d50-64203df5c3fd\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:39:46.857016 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:46.856768    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle podName:7117c313-6fb4-48b0-8f3e-16e2c9c8d551 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:50.856742212 +0000 UTC m=+50.293321034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle") pod "router-default-6bd44bd444-kdgsw" (UID: "7117c313-6fb4-48b0-8f3e-16e2c9c8d551") : configmap references non-existent config key: service-ca.crt
Sep 18 18:39:46.857016 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:46.856794    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 18:39:46.857016 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:46.856825    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 18:39:46.857016 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:46.856794    2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 18:39:46.857016 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:46.856845    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls podName:9b2a9f30-7931-4049-8d50-64203df5c3fd nodeName:}" failed. No retries permitted until 2026-09-18 18:39:50.85683078 +0000 UTC m=+50.293409620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-p27wv" (UID: "9b2a9f30-7931-4049-8d50-64203df5c3fd") : secret "cluster-monitoring-operator-tls" not found
Sep 18 18:39:46.857016 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:46.856874    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs podName:7117c313-6fb4-48b0-8f3e-16e2c9c8d551 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:50.856861507 +0000 UTC m=+50.293440327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs") pod "router-default-6bd44bd444-kdgsw" (UID: "7117c313-6fb4-48b0-8f3e-16e2c9c8d551") : secret "router-metrics-certs-default" not found
Sep 18 18:39:46.857016 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:46.856893    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert podName:f649578f-b8d4-4207-9dcc-69ce583fc6f6 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:50.856883146 +0000 UTC m=+50.293461967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4lcv7" (UID: "f649578f-b8d4-4207-9dcc-69ce583fc6f6") : secret "networking-console-plugin-cert" not found
Sep 18 18:39:47.058533 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:47.058448    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zqkl5\" (UID: \"51902e02-aefc-4f2a-bab5-b65c1f981852\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"
Sep 18 18:39:47.058688 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:47.058559    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 18:39:47.058688 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:47.058628    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls podName:51902e02-aefc-4f2a-bab5-b65c1f981852 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:51.058614352 +0000 UTC m=+50.495193172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-zqkl5" (UID: "51902e02-aefc-4f2a-bab5-b65c1f981852") : secret "samples-operator-tls" not found
Sep 18 18:39:49.803630 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:49.803594    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj" event={"ID":"8b41f87c-1c13-44ea-ae32-8ff22711e45e","Type":"ContainerStarted","Data":"9e2111e503da03a59744f086ab0213f661796373172f2e14ee1395487ba04215"}
Sep 18 18:39:49.805862 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:49.805838    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/0.log"
Sep 18 18:39:49.805993 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:49.805880    2558 generic.go:356] "Generic (PLEG): container finished" podID="8c3a44de-98c4-42e8-9ed2-7587293418ca" containerID="67adab1e14fa2475e892ff871d0fb0bfb984d681860363eeb8e4baccb1f5be4f" exitCode=255
Sep 18 18:39:49.805993 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:49.805954    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-whw2g" event={"ID":"8c3a44de-98c4-42e8-9ed2-7587293418ca","Type":"ContainerDied","Data":"67adab1e14fa2475e892ff871d0fb0bfb984d681860363eeb8e4baccb1f5be4f"}
Sep 18 18:39:49.806201 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:49.806171    2558 scope.go:117] "RemoveContainer" containerID="67adab1e14fa2475e892ff871d0fb0bfb984d681860363eeb8e4baccb1f5be4f"
Sep 18 18:39:49.808630 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:49.808329    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nd8ht" event={"ID":"3ff38d46-064b-4f8a-b34b-66a6f125e317","Type":"ContainerStarted","Data":"ba27ea901b4788714e15295d2a6d77102aa6f670fad579cc866f06a45a0f61d8"}
Sep 18 18:39:49.810727 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:49.810664    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-q64fh" event={"ID":"63302165-e325-4efd-88d8-dda368f6066e","Type":"ContainerStarted","Data":"25e250bb4ff7d4377d901d5384f8b21246d74395fce9423d314a14d0f43fea51"}
Sep 18 18:39:49.812623 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:49.812590    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr" event={"ID":"cccc2e3c-0d90-42aa-a136-069631e71567","Type":"ContainerStarted","Data":"801db0f4cd15ce9a50bd35785504c2b2dbf217198965262595014555943ccbd3"}
Sep 18 18:39:49.821293 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:49.821238    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj" podStartSLOduration=1.828334566 podStartE2EDuration="6.821217611s" totalImagesPullingTime="4.992883045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:49.820068739 +0000 UTC m=+49.256647581" watchObservedRunningTime="2026-09-18 18:39:49.821217611 +0000 UTC m=+49.257796456"
Sep 18 18:39:49.861128 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:49.856819    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-nd8ht" podStartSLOduration=1.895271658 podStartE2EDuration="6.85679766s" totalImagesPullingTime="4.961526002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:49.856228833 +0000 UTC m=+49.292807677" watchObservedRunningTime="2026-09-18 18:39:49.85679766 +0000 UTC m=+49.293376501"
Sep 18 18:39:49.876552 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:49.876498    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-q64fh" podStartSLOduration=1.9436458509999999 podStartE2EDuration="6.876478145s" totalImagesPullingTime="4.932832294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:49.876304247 +0000 UTC m=+49.312883090" watchObservedRunningTime="2026-09-18 18:39:49.876478145 +0000 UTC m=+49.313056988"
Sep 18 18:39:49.891645 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:49.891590    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr" podStartSLOduration=1.942652745 podStartE2EDuration="6.891569699s" totalImagesPullingTime="4.948916954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:49.890709344 +0000 UTC m=+49.327288189" watchObservedRunningTime="2026-09-18 18:39:49.891569699 +0000 UTC m=+49.328148536"
Sep 18 18:39:50.156412 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.156361    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wpbnx"]
Sep 18 18:39:50.179566 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.179536    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wpbnx"]
Sep 18 18:39:50.179717 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.179687    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.182495 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.182471    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 18:39:50.182495 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.182471    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 18:39:50.182657 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.182565    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-swhrv\""
Sep 18 18:39:50.185888 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.185863    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/629ab9c3-0c87-4f52-9770-c861b9ea5705-data-volume\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.185996 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.185911    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/629ab9c3-0c87-4f52-9770-c861b9ea5705-crio-socket\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.185996 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.185956    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c78w9\" (UniqueName: \"kubernetes.io/projected/629ab9c3-0c87-4f52-9770-c861b9ea5705-kube-api-access-c78w9\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.186118 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.185999    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/629ab9c3-0c87-4f52-9770-c861b9ea5705-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.186177 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.186145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.240466 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.240432    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-2jg78"]
Sep 18 18:39:50.263374 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.263348    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-2jg78"]
Sep 18 18:39:50.263374 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.263352    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-2jg78"
Sep 18 18:39:50.265831 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.265805    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 18:39:50.265961 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.265844    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 18:39:50.265961 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.265903    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dgffh\""
Sep 18 18:39:50.286552 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.286528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.286657 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.286578    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/629ab9c3-0c87-4f52-9770-c861b9ea5705-data-volume\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.286701 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.286675    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:39:50.286701 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.286688    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/629ab9c3-0c87-4f52-9770-c861b9ea5705-crio-socket\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.286771 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.286716    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c78w9\" (UniqueName: \"kubernetes.io/projected/629ab9c3-0c87-4f52-9770-c861b9ea5705-kube-api-access-c78w9\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.286771 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.286756    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls podName:629ab9c3-0c87-4f52-9770-c861b9ea5705 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:50.786730058 +0000 UTC m=+50.223308879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wpbnx" (UID: "629ab9c3-0c87-4f52-9770-c861b9ea5705") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:39:50.286865 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.286793    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/629ab9c3-0c87-4f52-9770-c861b9ea5705-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.286865 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.286826    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/629ab9c3-0c87-4f52-9770-c861b9ea5705-data-volume\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.286949 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.286882    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 18:39:50.286949 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.286913    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/629ab9c3-0c87-4f52-9770-c861b9ea5705-kube-rbac-proxy-cm podName:629ab9c3-0c87-4f52-9770-c861b9ea5705 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:50.786904889 +0000 UTC m=+50.223483710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/629ab9c3-0c87-4f52-9770-c861b9ea5705-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wpbnx" (UID: "629ab9c3-0c87-4f52-9770-c861b9ea5705") : configmap "kube-rbac-proxy" not found
Sep 18 18:39:50.286949 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.286923    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db6qt\" (UniqueName: \"kubernetes.io/projected/4019fadb-4f88-4e5d-a1c6-51f823d9e3a5-kube-api-access-db6qt\") pod \"migrator-76794cbd6c-2jg78\" (UID: \"4019fadb-4f88-4e5d-a1c6-51f823d9e3a5\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-2jg78"
Sep 18 18:39:50.286949 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.286933    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/629ab9c3-0c87-4f52-9770-c861b9ea5705-crio-socket\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.295392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.295373    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c78w9\" (UniqueName: \"kubernetes.io/projected/629ab9c3-0c87-4f52-9770-c861b9ea5705-kube-api-access-c78w9\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.388172 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.388141    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db6qt\" (UniqueName: \"kubernetes.io/projected/4019fadb-4f88-4e5d-a1c6-51f823d9e3a5-kube-api-access-db6qt\") pod \"migrator-76794cbd6c-2jg78\" (UID: \"4019fadb-4f88-4e5d-a1c6-51f823d9e3a5\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-2jg78"
Sep 18 18:39:50.395770 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.395746    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db6qt\" (UniqueName: \"kubernetes.io/projected/4019fadb-4f88-4e5d-a1c6-51f823d9e3a5-kube-api-access-db6qt\") pod \"migrator-76794cbd6c-2jg78\" (UID: \"4019fadb-4f88-4e5d-a1c6-51f823d9e3a5\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-2jg78"
Sep 18 18:39:50.571678 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.571610    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-2jg78"
Sep 18 18:39:50.790532 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.790499    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.790660 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.790572    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/629ab9c3-0c87-4f52-9770-c861b9ea5705-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:50.790660 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.790652    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:39:50.790740 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.790698    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 18:39:50.790740 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.790714    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls podName:629ab9c3-0c87-4f52-9770-c861b9ea5705 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:51.790700103 +0000 UTC m=+51.227278924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wpbnx" (UID: "629ab9c3-0c87-4f52-9770-c861b9ea5705") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:39:50.790813 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.790753    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/629ab9c3-0c87-4f52-9770-c861b9ea5705-kube-rbac-proxy-cm podName:629ab9c3-0c87-4f52-9770-c861b9ea5705 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:51.790735042 +0000 UTC m=+51.227313862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/629ab9c3-0c87-4f52-9770-c861b9ea5705-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wpbnx" (UID: "629ab9c3-0c87-4f52-9770-c861b9ea5705") : configmap "kube-rbac-proxy" not found
Sep 18 18:39:50.817513 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.817491    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 18:39:50.817906 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.817874    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/0.log"
Sep 18 18:39:50.817953 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.817903    2558 generic.go:356] "Generic (PLEG): container finished" podID="8c3a44de-98c4-42e8-9ed2-7587293418ca" containerID="179a7016484614db1f2e2277cb67462d75b6f5bf3dff5a82fe39fb920d233679" exitCode=255
Sep 18 18:39:50.817997 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.817943    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-whw2g" event={"ID":"8c3a44de-98c4-42e8-9ed2-7587293418ca","Type":"ContainerDied","Data":"179a7016484614db1f2e2277cb67462d75b6f5bf3dff5a82fe39fb920d233679"}
Sep 18 18:39:50.818038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.818011    2558 scope.go:117] "RemoveContainer" containerID="67adab1e14fa2475e892ff871d0fb0bfb984d681860363eeb8e4baccb1f5be4f"
Sep 18 18:39:50.818281 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.818264    2558 scope.go:117] "RemoveContainer" containerID="179a7016484614db1f2e2277cb67462d75b6f5bf3dff5a82fe39fb920d233679"
Sep 18 18:39:50.818469 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.818447    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-whw2g_openshift-console-operator(8c3a44de-98c4-42e8-9ed2-7587293418ca)\"" pod="openshift-console-operator/console-operator-fd4bb46c-whw2g" podUID="8c3a44de-98c4-42e8-9ed2-7587293418ca"
Sep 18 18:39:50.858623 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.858597    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-2jg78"]
Sep 18 18:39:50.861796 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:50.861773    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4019fadb_4f88_4e5d_a1c6_51f823d9e3a5.slice/crio-f349b18d365530753bdf1d9911db2756bd3d6215c15c19565beb9657c5f53a17 WatchSource:0}: Error finding container f349b18d365530753bdf1d9911db2756bd3d6215c15c19565beb9657c5f53a17: Status 404 returned error can't find the container with id f349b18d365530753bdf1d9911db2756bd3d6215c15c19565beb9657c5f53a17
Sep 18 18:39:50.892312 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.891738    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:50.892312 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.891830    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4lcv7\" (UID: \"f649578f-b8d4-4207-9dcc-69ce583fc6f6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"
Sep 18 18:39:50.892312 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.891917    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:50.892312 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.891942    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle podName:7117c313-6fb4-48b0-8f3e-16e2c9c8d551 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:58.891920418 +0000 UTC m=+58.328499241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle") pod "router-default-6bd44bd444-kdgsw" (UID: "7117c313-6fb4-48b0-8f3e-16e2c9c8d551") : configmap references non-existent config key: service-ca.crt
Sep 18 18:39:50.892312 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.891963    2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 18:39:50.892312 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.891997    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 18:39:50.892312 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.892003    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert podName:f649578f-b8d4-4207-9dcc-69ce583fc6f6 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:58.891991494 +0000 UTC m=+58.328570329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4lcv7" (UID: "f649578f-b8d4-4207-9dcc-69ce583fc6f6") : secret "networking-console-plugin-cert" not found
Sep 18 18:39:50.892312 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.892042    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs podName:7117c313-6fb4-48b0-8f3e-16e2c9c8d551 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:58.892027326 +0000 UTC m=+58.328606169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs") pod "router-default-6bd44bd444-kdgsw" (UID: "7117c313-6fb4-48b0-8f3e-16e2c9c8d551") : secret "router-metrics-certs-default" not found
Sep 18 18:39:50.892312 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:50.892108    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-p27wv\" (UID: \"9b2a9f30-7931-4049-8d50-64203df5c3fd\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:39:50.892312 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.892151    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 18:39:50.892312 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:50.892269    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls podName:9b2a9f30-7931-4049-8d50-64203df5c3fd nodeName:}" failed. No retries permitted until 2026-09-18 18:39:58.892245865 +0000 UTC m=+58.328824690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-p27wv" (UID: "9b2a9f30-7931-4049-8d50-64203df5c3fd") : secret "cluster-monitoring-operator-tls" not found
Sep 18 18:39:51.094118 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:51.094075    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zqkl5\" (UID: \"51902e02-aefc-4f2a-bab5-b65c1f981852\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"
Sep 18 18:39:51.094286 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:51.094215    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 18:39:51.094286 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:51.094270    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls podName:51902e02-aefc-4f2a-bab5-b65c1f981852 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:59.094256929 +0000 UTC m=+58.530835762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-zqkl5" (UID: "51902e02-aefc-4f2a-bab5-b65c1f981852") : secret "samples-operator-tls" not found
Sep 18 18:39:51.527548 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:51.527523    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xbz24_aa59b9bc-694f-4b16-99bf-5d31a85bca74/dns-node-resolver/0.log"
Sep 18 18:39:51.801083 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:51.801005    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:51.801257 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:51.801153    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:39:51.801257 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:51.801178    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/629ab9c3-0c87-4f52-9770-c861b9ea5705-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:51.801257 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:51.801235    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls podName:629ab9c3-0c87-4f52-9770-c861b9ea5705 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:53.801219805 +0000 UTC m=+53.237798630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wpbnx" (UID: "629ab9c3-0c87-4f52-9770-c861b9ea5705") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:39:51.801401 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:51.801272    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 18:39:51.801401 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:51.801321    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/629ab9c3-0c87-4f52-9770-c861b9ea5705-kube-rbac-proxy-cm podName:629ab9c3-0c87-4f52-9770-c861b9ea5705 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:53.801310389 +0000 UTC m=+53.237889233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/629ab9c3-0c87-4f52-9770-c861b9ea5705-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wpbnx" (UID: "629ab9c3-0c87-4f52-9770-c861b9ea5705") : configmap "kube-rbac-proxy" not found
Sep 18 18:39:51.823085 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:51.823054    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 18:39:51.823550 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:51.823414    2558 scope.go:117] "RemoveContainer" containerID="179a7016484614db1f2e2277cb67462d75b6f5bf3dff5a82fe39fb920d233679"
Sep 18 18:39:51.823618 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:51.823581    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-whw2g_openshift-console-operator(8c3a44de-98c4-42e8-9ed2-7587293418ca)\"" pod="openshift-console-operator/console-operator-fd4bb46c-whw2g" podUID="8c3a44de-98c4-42e8-9ed2-7587293418ca"
Sep 18 18:39:51.824421 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:51.824398    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-2jg78" event={"ID":"4019fadb-4f88-4e5d-a1c6-51f823d9e3a5","Type":"ContainerStarted","Data":"f349b18d365530753bdf1d9911db2756bd3d6215c15c19565beb9657c5f53a17"}
Sep 18 18:39:52.154251 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.154213    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-p44lh"]
Sep 18 18:39:52.169124 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.169098    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-p44lh"]
Sep 18 18:39:52.169263 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.169223    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-p44lh"
Sep 18 18:39:52.171552 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.171525    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 18:39:52.171685 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.171572    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 18:39:52.171685 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.171572    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 18:39:52.171805 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.171689    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-n8fz2\""
Sep 18 18:39:52.172371 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.172342    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 18:39:52.205247 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.205226    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltkmr\" (UniqueName: \"kubernetes.io/projected/0ec50589-27bf-4fd6-8e45-f313d391b007-kube-api-access-ltkmr\") pod \"service-ca-55f6fc7b86-p44lh\" (UID: \"0ec50589-27bf-4fd6-8e45-f313d391b007\") " pod="openshift-service-ca/service-ca-55f6fc7b86-p44lh"
Sep 18 18:39:52.205345 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.205270    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0ec50589-27bf-4fd6-8e45-f313d391b007-signing-key\") pod \"service-ca-55f6fc7b86-p44lh\" (UID: \"0ec50589-27bf-4fd6-8e45-f313d391b007\") " pod="openshift-service-ca/service-ca-55f6fc7b86-p44lh"
Sep 18 18:39:52.205345 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.205287    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0ec50589-27bf-4fd6-8e45-f313d391b007-signing-cabundle\") pod \"service-ca-55f6fc7b86-p44lh\" (UID: \"0ec50589-27bf-4fd6-8e45-f313d391b007\") " pod="openshift-service-ca/service-ca-55f6fc7b86-p44lh"
Sep 18 18:39:52.306367 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.306338    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltkmr\" (UniqueName: \"kubernetes.io/projected/0ec50589-27bf-4fd6-8e45-f313d391b007-kube-api-access-ltkmr\") pod \"service-ca-55f6fc7b86-p44lh\" (UID: \"0ec50589-27bf-4fd6-8e45-f313d391b007\") " pod="openshift-service-ca/service-ca-55f6fc7b86-p44lh"
Sep 18 18:39:52.306488 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.306381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0ec50589-27bf-4fd6-8e45-f313d391b007-signing-key\") pod \"service-ca-55f6fc7b86-p44lh\" (UID: \"0ec50589-27bf-4fd6-8e45-f313d391b007\") " pod="openshift-service-ca/service-ca-55f6fc7b86-p44lh"
Sep 18 18:39:52.306488 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.306404    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0ec50589-27bf-4fd6-8e45-f313d391b007-signing-cabundle\") pod \"service-ca-55f6fc7b86-p44lh\" (UID: \"0ec50589-27bf-4fd6-8e45-f313d391b007\") " pod="openshift-service-ca/service-ca-55f6fc7b86-p44lh"
Sep 18 18:39:52.307001 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.306977    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0ec50589-27bf-4fd6-8e45-f313d391b007-signing-cabundle\") pod \"service-ca-55f6fc7b86-p44lh\" (UID: \"0ec50589-27bf-4fd6-8e45-f313d391b007\") " pod="openshift-service-ca/service-ca-55f6fc7b86-p44lh"
Sep 18 18:39:52.308882 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.308864    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0ec50589-27bf-4fd6-8e45-f313d391b007-signing-key\") pod \"service-ca-55f6fc7b86-p44lh\" (UID: \"0ec50589-27bf-4fd6-8e45-f313d391b007\") " pod="openshift-service-ca/service-ca-55f6fc7b86-p44lh"
Sep 18 18:39:52.315432 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.315406    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltkmr\" (UniqueName: \"kubernetes.io/projected/0ec50589-27bf-4fd6-8e45-f313d391b007-kube-api-access-ltkmr\") pod \"service-ca-55f6fc7b86-p44lh\" (UID: \"0ec50589-27bf-4fd6-8e45-f313d391b007\") " pod="openshift-service-ca/service-ca-55f6fc7b86-p44lh"
Sep 18 18:39:52.478155 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.477997    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-p44lh"
Sep 18 18:39:52.531775 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.531750    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zmgsq_aad288b6-deba-464d-917f-dcc47d794865/node-ca/0.log"
Sep 18 18:39:52.609324 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.609299    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:39:52.609437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.609416    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:39:52.609481 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:52.609437    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:39:52.609515 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:52.609488    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls podName:6455458b-83a0-4f35-857a-f11c3866299e nodeName:}" failed. No retries permitted until 2026-09-18 18:40:08.609471536 +0000 UTC m=+68.046050381 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls") pod "dns-default-zw2cl" (UID: "6455458b-83a0-4f35-857a-f11c3866299e") : secret "dns-default-metrics-tls" not found
Sep 18 18:39:52.609554 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:52.609524    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 18:39:52.609554 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:52.609533    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdbd487b5-zx747: secret "image-registry-tls" not found
Sep 18 18:39:52.609610 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:52.609570    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls podName:92f09b3c-b43d-4de1-b12f-b1a1ae026f41 nodeName:}" failed. No retries permitted until 2026-09-18 18:40:08.609559715 +0000 UTC m=+68.046138535 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls") pod "image-registry-7bdbd487b5-zx747" (UID: "92f09b3c-b43d-4de1-b12f-b1a1ae026f41") : secret "image-registry-tls" not found
Sep 18 18:39:52.773448 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.773376    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-p44lh"]
Sep 18 18:39:52.778570 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:39:52.778541    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec50589_27bf_4fd6_8e45_f313d391b007.slice/crio-d423d19a01c12830feded6eeb4a6bb4b67977c9c9ba36cdf62e43d0dfe5f2b24 WatchSource:0}: Error finding container d423d19a01c12830feded6eeb4a6bb4b67977c9c9ba36cdf62e43d0dfe5f2b24: Status 404 returned error can't find the container with id d423d19a01c12830feded6eeb4a6bb4b67977c9c9ba36cdf62e43d0dfe5f2b24
Sep 18 18:39:52.828761 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.828733    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-2jg78" event={"ID":"4019fadb-4f88-4e5d-a1c6-51f823d9e3a5","Type":"ContainerStarted","Data":"2ca2fa2f05fa3b46fccf2a61c2658ab6050588fdf9474bb21893d27eab713b79"}
Sep 18 18:39:52.829101 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.828793    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-2jg78" event={"ID":"4019fadb-4f88-4e5d-a1c6-51f823d9e3a5","Type":"ContainerStarted","Data":"fb0d2e109efa9f2522243363f4817e4c2be385dc70feee4d4d5620781875e148"}
Sep 18 18:39:52.830163 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.830135    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-p44lh" event={"ID":"0ec50589-27bf-4fd6-8e45-f313d391b007","Type":"ContainerStarted","Data":"d423d19a01c12830feded6eeb4a6bb4b67977c9c9ba36cdf62e43d0dfe5f2b24"}
Sep 18 18:39:52.848839 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.848796    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-2jg78" podStartSLOduration=1.233417618 podStartE2EDuration="2.848777515s" totalImagesPullingTime="1.615359897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:52.848472869 +0000 UTC m=+52.285051710" watchObservedRunningTime="2026-09-18 18:39:52.848777515 +0000 UTC m=+52.285356358"
Sep 18 18:39:52.927278 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:52.927248    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-42z88_bf18e0b9-6a23-4338-893f-e8d7ca463c1e/serve-healthcheck-canary/0.log"
Sep 18 18:39:53.398169 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:53.398134    2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:53.398339 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:53.398233    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:39:53.398579 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:53.398566    2558 scope.go:117] "RemoveContainer" containerID="179a7016484614db1f2e2277cb67462d75b6f5bf3dff5a82fe39fb920d233679"
Sep 18 18:39:53.398728 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:53.398708    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-whw2g_openshift-console-operator(8c3a44de-98c4-42e8-9ed2-7587293418ca)\"" pod="openshift-console-operator/console-operator-fd4bb46c-whw2g" podUID="8c3a44de-98c4-42e8-9ed2-7587293418ca"
Sep 18 18:39:53.821811 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:53.821714    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/629ab9c3-0c87-4f52-9770-c861b9ea5705-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:53.821973 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:53.821883    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:53.822026 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:53.822001    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:39:53.822074 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:53.822062    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls podName:629ab9c3-0c87-4f52-9770-c861b9ea5705 nodeName:}" failed. No retries permitted until 2026-09-18 18:39:57.822044201 +0000 UTC m=+57.258623024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wpbnx" (UID: "629ab9c3-0c87-4f52-9770-c861b9ea5705") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:39:53.822803 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:53.822770    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/629ab9c3-0c87-4f52-9770-c861b9ea5705-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:53.836936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:53.836881    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-p44lh" event={"ID":"0ec50589-27bf-4fd6-8e45-f313d391b007","Type":"ContainerStarted","Data":"60cb8e18f4e1b91d7965ed2198616c762f8ad86dcdd0ace6d303ce1c2a35813d"}
Sep 18 18:39:53.854727 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:53.854679    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-p44lh" podStartSLOduration=1.8546616889999998 podStartE2EDuration="1.854661689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:39:53.85346083 +0000 UTC m=+53.290039673" watchObservedRunningTime="2026-09-18 18:39:53.854661689 +0000 UTC m=+53.291240533"
Sep 18 18:39:57.859413 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:57.859382    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:39:57.859969 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:57.859558    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:39:57.859969 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:57.859631    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls podName:629ab9c3-0c87-4f52-9770-c861b9ea5705 nodeName:}" failed. No retries permitted until 2026-09-18 18:40:05.859611684 +0000 UTC m=+65.296190519 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wpbnx" (UID: "629ab9c3-0c87-4f52-9770-c861b9ea5705") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:39:58.969582 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:58.969548    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:58.969999 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:58.969597    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4lcv7\" (UID: \"f649578f-b8d4-4207-9dcc-69ce583fc6f6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"
Sep 18 18:39:58.969999 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:58.969626    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:58.969999 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:58.969723    2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 18:39:58.969999 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:58.969801    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle podName:7117c313-6fb4-48b0-8f3e-16e2c9c8d551 nodeName:}" failed. No retries permitted until 2026-09-18 18:40:14.969778521 +0000 UTC m=+74.406357348 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle") pod "router-default-6bd44bd444-kdgsw" (UID: "7117c313-6fb4-48b0-8f3e-16e2c9c8d551") : configmap references non-existent config key: service-ca.crt
Sep 18 18:39:58.969999 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:58.969830    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert podName:f649578f-b8d4-4207-9dcc-69ce583fc6f6 nodeName:}" failed. No retries permitted until 2026-09-18 18:40:14.969818873 +0000 UTC m=+74.406397695 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-4lcv7" (UID: "f649578f-b8d4-4207-9dcc-69ce583fc6f6") : secret "networking-console-plugin-cert" not found
Sep 18 18:39:58.969999 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:58.969897    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-p27wv\" (UID: \"9b2a9f30-7931-4049-8d50-64203df5c3fd\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:39:58.970297 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:58.970017    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 18:39:58.970297 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:39:58.970063    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls podName:9b2a9f30-7931-4049-8d50-64203df5c3fd nodeName:}" failed. No retries permitted until 2026-09-18 18:40:14.970049959 +0000 UTC m=+74.406628779 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-p27wv" (UID: "9b2a9f30-7931-4049-8d50-64203df5c3fd") : secret "cluster-monitoring-operator-tls" not found
Sep 18 18:39:58.972073 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:58.972046    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-metrics-certs\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:39:59.172296 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:59.172258    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zqkl5\" (UID: \"51902e02-aefc-4f2a-bab5-b65c1f981852\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"
Sep 18 18:39:59.174773 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:59.174743    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/51902e02-aefc-4f2a-bab5-b65c1f981852-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-zqkl5\" (UID: \"51902e02-aefc-4f2a-bab5-b65c1f981852\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"
Sep 18 18:39:59.181621 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:59.181602    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"
Sep 18 18:39:59.477033 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:59.476992    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5"]
Sep 18 18:39:59.857610 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:39:59.857578    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5" event={"ID":"51902e02-aefc-4f2a-bab5-b65c1f981852","Type":"ContainerStarted","Data":"e29a074044e57fef7e81894553914fea90454f6d9476866320926c89693f3152"}
Sep 18 18:40:02.730486 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:02.730459    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l6vdn"
Sep 18 18:40:02.868684 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:02.868651    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5" event={"ID":"51902e02-aefc-4f2a-bab5-b65c1f981852","Type":"ContainerStarted","Data":"d5bfb75918de318293ea996493607da7e572feb687f2970b0089560883b3f682"}
Sep 18 18:40:02.868849 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:02.868706    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5" event={"ID":"51902e02-aefc-4f2a-bab5-b65c1f981852","Type":"ContainerStarted","Data":"21baf517ae6493dae81bbebe6f9bc560196975c26204f98c0f93fe709d560df2"}
Sep 18 18:40:02.882852 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:02.882801    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-zqkl5" podStartSLOduration=17.473558693 podStartE2EDuration="19.882783317s" totalImagesPullingTime="2.409224624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:40:02.882217782 +0000 UTC m=+62.318796625" watchObservedRunningTime="2026-09-18 18:40:02.882783317 +0000 UTC m=+62.319362160"
Sep 18 18:40:05.938688 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:05.938648    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:40:05.941027 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:05.941005    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/629ab9c3-0c87-4f52-9770-c861b9ea5705-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wpbnx\" (UID: \"629ab9c3-0c87-4f52-9770-c861b9ea5705\") " pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:40:06.091748 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:06.091711    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-swhrv\""
Sep 18 18:40:06.100579 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:06.100562    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wpbnx"
Sep 18 18:40:06.368972 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:06.368940    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wpbnx"]
Sep 18 18:40:06.371909 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:40:06.371881    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629ab9c3_0c87_4f52_9770_c861b9ea5705.slice/crio-58a67560ae995dddea1ee7b4ea77eacafe5c0e61b61c1655fe1ae1232576bd98 WatchSource:0}: Error finding container 58a67560ae995dddea1ee7b4ea77eacafe5c0e61b61c1655fe1ae1232576bd98: Status 404 returned error can't find the container with id 58a67560ae995dddea1ee7b4ea77eacafe5c0e61b61c1655fe1ae1232576bd98
Sep 18 18:40:06.884277 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:06.884243    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wpbnx" event={"ID":"629ab9c3-0c87-4f52-9770-c861b9ea5705","Type":"ContainerStarted","Data":"8f65c0074077157e5c90818f8a5dc96c5d01c60c81c1db9efc061220eb828158"}
Sep 18 18:40:06.884440 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:06.884300    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wpbnx" event={"ID":"629ab9c3-0c87-4f52-9770-c861b9ea5705","Type":"ContainerStarted","Data":"58a67560ae995dddea1ee7b4ea77eacafe5c0e61b61c1655fe1ae1232576bd98"}
Sep 18 18:40:07.569586 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:07.569558    2558 scope.go:117] "RemoveContainer" containerID="179a7016484614db1f2e2277cb67462d75b6f5bf3dff5a82fe39fb920d233679"
Sep 18 18:40:07.889848 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:07.889822    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 18:40:07.890016 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:07.889938    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-whw2g" event={"ID":"8c3a44de-98c4-42e8-9ed2-7587293418ca","Type":"ContainerStarted","Data":"888b4b718bd05c5f9deafa4be6e719a2c52af5c6d1e17ee061074dce80ddf40a"}
Sep 18 18:40:07.890260 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:07.890233    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:40:07.891927 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:07.891904    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wpbnx" event={"ID":"629ab9c3-0c87-4f52-9770-c861b9ea5705","Type":"ContainerStarted","Data":"541b8297e246708f25840bced012902961b5e01739c42534100579799ce5375e"}
Sep 18 18:40:07.905028 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:07.904989    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-whw2g" podStartSLOduration=19.827448991 podStartE2EDuration="24.904977118s" totalImagesPullingTime="5.077528127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:40:07.904019533 +0000 UTC m=+67.340598377" watchObservedRunningTime="2026-09-18 18:40:07.904977118 +0000 UTC m=+67.341555999"
Sep 18 18:40:08.358383 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.358351    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs\") pod \"network-metrics-daemon-dtwsf\" (UID: \"2d9a33e8-d5ff-482e-b822-5aa13b177a6f\") " pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:40:08.358579 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.358447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:40:08.360831 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.360805    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 18:40:08.360964 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.360852    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 18:40:08.371176 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.371154    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ac6087e-296b-4917-bee6-8540193b8719-original-pull-secret\") pod \"global-pull-secret-syncer-w9bz8\" (UID: \"9ac6087e-296b-4917-bee6-8540193b8719\") " pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:40:08.371303 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.371219    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9a33e8-d5ff-482e-b822-5aa13b177a6f-metrics-certs\") pod \"network-metrics-daemon-dtwsf\" (UID: \"2d9a33e8-d5ff-482e-b822-5aa13b177a6f\") " pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:40:08.459532 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.459487    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrh7\" (UniqueName: \"kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7\") pod \"network-check-target-xrssq\" (UID: \"d2997254-b30e-438d-b105-40088a498ee5\") " pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:40:08.462727 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.462698    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znrh7\" (UniqueName: \"kubernetes.io/projected/d2997254-b30e-438d-b105-40088a498ee5-kube-api-access-znrh7\") pod \"network-check-target-xrssq\" (UID: \"d2997254-b30e-438d-b105-40088a498ee5\") " pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:40:08.474777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.474747    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-whw2g"
Sep 18 18:40:08.583808 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.583779    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5mb7g\""
Sep 18 18:40:08.591972 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.591947    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtwsf"
Sep 18 18:40:08.593123 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.592932    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kbgtj\""
Sep 18 18:40:08.597110 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.597092    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w9bz8"
Sep 18 18:40:08.601953 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.601904    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:40:08.662122 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.662043    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:40:08.662285 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.662135    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:40:08.664820 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.664786    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6455458b-83a0-4f35-857a-f11c3866299e-metrics-tls\") pod \"dns-default-zw2cl\" (UID: \"6455458b-83a0-4f35-857a-f11c3866299e\") " pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:40:08.664959 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.664888    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls\") pod \"image-registry-7bdbd487b5-zx747\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") " pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:40:08.826353 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.826318    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dvnsn\""
Sep 18 18:40:08.834445 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.834402    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:40:08.845019 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.844974    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r76nj\""
Sep 18 18:40:08.852998 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:08.852980    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:40:09.171591 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.171561    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dtwsf"]
Sep 18 18:40:09.174557 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:40:09.174526    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9a33e8_d5ff_482e_b822_5aa13b177a6f.slice/crio-eb3e4bbb06b726869a1eb0dcc21b7b96719060625a6d4ad65f33f78d07f52b56 WatchSource:0}: Error finding container eb3e4bbb06b726869a1eb0dcc21b7b96719060625a6d4ad65f33f78d07f52b56: Status 404 returned error can't find the container with id eb3e4bbb06b726869a1eb0dcc21b7b96719060625a6d4ad65f33f78d07f52b56
Sep 18 18:40:09.174917 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.174898    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bdbd487b5-zx747"]
Sep 18 18:40:09.178392 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:40:09.178369    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f09b3c_b43d_4de1_b12f_b1a1ae026f41.slice/crio-1f6281c549a46cba6fb35c29a8ede46f155f13f344389ce84dd6fbe1bc9306a7 WatchSource:0}: Error finding container 1f6281c549a46cba6fb35c29a8ede46f155f13f344389ce84dd6fbe1bc9306a7: Status 404 returned error can't find the container with id 1f6281c549a46cba6fb35c29a8ede46f155f13f344389ce84dd6fbe1bc9306a7
Sep 18 18:40:09.181718 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.181694    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w9bz8"]
Sep 18 18:40:09.184917 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:40:09.184896    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac6087e_296b_4917_bee6_8540193b8719.slice/crio-68400e9817aad6f65536c798156bb9829352c5a514d175723badfdfb733a3f6c WatchSource:0}: Error finding container 68400e9817aad6f65536c798156bb9829352c5a514d175723badfdfb733a3f6c: Status 404 returned error can't find the container with id 68400e9817aad6f65536c798156bb9829352c5a514d175723badfdfb733a3f6c
Sep 18 18:40:09.195974 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.195954    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zw2cl"]
Sep 18 18:40:09.198534 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.198513    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xrssq"]
Sep 18 18:40:09.200605 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:40:09.200583    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6455458b_83a0_4f35_857a_f11c3866299e.slice/crio-b86c32ba38eb96d1e83030a3b8d98fdc52d25b8cac861135f035cd2affc24eb9 WatchSource:0}: Error finding container b86c32ba38eb96d1e83030a3b8d98fdc52d25b8cac861135f035cd2affc24eb9: Status 404 returned error can't find the container with id b86c32ba38eb96d1e83030a3b8d98fdc52d25b8cac861135f035cd2affc24eb9
Sep 18 18:40:09.207468 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:40:09.207447    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2997254_b30e_438d_b105_40088a498ee5.slice/crio-6ced94b3d14dd40d8d1b0480fc4a9890e7322edf0c0e7dc1df138f2a6bb7a077 WatchSource:0}: Error finding container 6ced94b3d14dd40d8d1b0480fc4a9890e7322edf0c0e7dc1df138f2a6bb7a077: Status 404 returned error can't find the container with id 6ced94b3d14dd40d8d1b0480fc4a9890e7322edf0c0e7dc1df138f2a6bb7a077
Sep 18 18:40:09.904158 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.904114    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xrssq" event={"ID":"d2997254-b30e-438d-b105-40088a498ee5","Type":"ContainerStarted","Data":"da841acf9a1ed081d99b737ef6288c4cd70d7e80ba0f1bea9c6f45bbfb00a089"}
Sep 18 18:40:09.904694 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.904177    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xrssq" event={"ID":"d2997254-b30e-438d-b105-40088a498ee5","Type":"ContainerStarted","Data":"6ced94b3d14dd40d8d1b0480fc4a9890e7322edf0c0e7dc1df138f2a6bb7a077"}
Sep 18 18:40:09.904928 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.904907    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:40:09.906791 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.906735    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zw2cl" event={"ID":"6455458b-83a0-4f35-857a-f11c3866299e","Type":"ContainerStarted","Data":"b86c32ba38eb96d1e83030a3b8d98fdc52d25b8cac861135f035cd2affc24eb9"}
Sep 18 18:40:09.910887 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.910301    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wpbnx" event={"ID":"629ab9c3-0c87-4f52-9770-c861b9ea5705","Type":"ContainerStarted","Data":"52925f19306c55e0af7631a4ac057f476b9bf633af63dd541d13257ad0ea01ca"}
Sep 18 18:40:09.912515 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.912465    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dtwsf" event={"ID":"2d9a33e8-d5ff-482e-b822-5aa13b177a6f","Type":"ContainerStarted","Data":"eb3e4bbb06b726869a1eb0dcc21b7b96719060625a6d4ad65f33f78d07f52b56"}
Sep 18 18:40:09.914865 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.914814    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bdbd487b5-zx747" event={"ID":"92f09b3c-b43d-4de1-b12f-b1a1ae026f41","Type":"ContainerStarted","Data":"9943ec032b40983069cf9b4068faf4b4054073df1174111311fa0a63d806facd"}
Sep 18 18:40:09.914865 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.914850    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bdbd487b5-zx747" event={"ID":"92f09b3c-b43d-4de1-b12f-b1a1ae026f41","Type":"ContainerStarted","Data":"1f6281c549a46cba6fb35c29a8ede46f155f13f344389ce84dd6fbe1bc9306a7"}
Sep 18 18:40:09.915035 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.914879    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:40:09.917284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.916876    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w9bz8" event={"ID":"9ac6087e-296b-4917-bee6-8540193b8719","Type":"ContainerStarted","Data":"68400e9817aad6f65536c798156bb9829352c5a514d175723badfdfb733a3f6c"}
Sep 18 18:40:09.920709 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.920658    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xrssq" podStartSLOduration=67.920641706 podStartE2EDuration="1m7.920641706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:40:09.918491651 +0000 UTC m=+69.355070494" watchObservedRunningTime="2026-09-18 18:40:09.920641706 +0000 UTC m=+69.357220553"
Sep 18 18:40:09.957275 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.957230    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wpbnx" podStartSLOduration=17.488581722 podStartE2EDuration="19.957214277s" totalImagesPullingTime="2.468632555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:39:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:40:09.955987387 +0000 UTC m=+69.392566228" watchObservedRunningTime="2026-09-18 18:40:09.957214277 +0000 UTC m=+69.393793109"
Sep 18 18:40:09.957589 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:09.957565    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7bdbd487b5-zx747" podStartSLOduration=66.9575586 podStartE2EDuration="1m6.9575586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:40:09.940434297 +0000 UTC m=+69.377013140" watchObservedRunningTime="2026-09-18 18:40:09.9575586 +0000 UTC m=+69.394137442"
Sep 18 18:40:10.922710 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:10.922667    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dtwsf" event={"ID":"2d9a33e8-d5ff-482e-b822-5aa13b177a6f","Type":"ContainerStarted","Data":"8b767ccff9de7a52fc66cafd74e37de2cbc32aef26bd65021dd6dd6d44574dcc"}
Sep 18 18:40:10.923234 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:10.922737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dtwsf" event={"ID":"2d9a33e8-d5ff-482e-b822-5aa13b177a6f","Type":"ContainerStarted","Data":"f3892caa097052474849eb77d8a6ad73aeb1362b1e9166dd9a1522d8e1ceca6d"}
Sep 18 18:40:10.937715 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:10.937669    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dtwsf" podStartSLOduration=67.801128799 podStartE2EDuration="1m8.937654707s" totalImagesPullingTime="1.136525908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:40:10.937369835 +0000 UTC m=+70.373948679" watchObservedRunningTime="2026-09-18 18:40:10.937654707 +0000 UTC m=+70.374233549"
Sep 18 18:40:11.581699 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:11.581669    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-rk7wk"]
Sep 18 18:40:11.585252 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:11.585233    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-rk7wk"
Sep 18 18:40:11.587868 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:11.587676    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 18:40:11.588097 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:11.588081    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-2rgjd\""
Sep 18 18:40:11.588473 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:11.588250    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 18:40:11.597399 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:11.597337    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-rk7wk"]
Sep 18 18:40:11.693740 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:11.693710    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwm6n\" (UniqueName: \"kubernetes.io/projected/0801c387-848b-4c2f-8188-97b7720c46d8-kube-api-access-qwm6n\") pod \"downloads-6594f4c75-rk7wk\" (UID: \"0801c387-848b-4c2f-8188-97b7720c46d8\") " pod="openshift-console/downloads-6594f4c75-rk7wk"
Sep 18 18:40:11.794500 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:11.794417    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwm6n\" (UniqueName: \"kubernetes.io/projected/0801c387-848b-4c2f-8188-97b7720c46d8-kube-api-access-qwm6n\") pod \"downloads-6594f4c75-rk7wk\" (UID: \"0801c387-848b-4c2f-8188-97b7720c46d8\") " pod="openshift-console/downloads-6594f4c75-rk7wk"
Sep 18 18:40:11.803064 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:11.803036    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwm6n\" (UniqueName: \"kubernetes.io/projected/0801c387-848b-4c2f-8188-97b7720c46d8-kube-api-access-qwm6n\") pod \"downloads-6594f4c75-rk7wk\" (UID: \"0801c387-848b-4c2f-8188-97b7720c46d8\") " pod="openshift-console/downloads-6594f4c75-rk7wk"
Sep 18 18:40:11.906112 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:11.906082    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-rk7wk"
Sep 18 18:40:11.930163 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:11.930134    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zw2cl" event={"ID":"6455458b-83a0-4f35-857a-f11c3866299e","Type":"ContainerStarted","Data":"b5ee19d7704699dbbeefe0397488dad47391b3905cec15d28297f3d37c7646e7"}
Sep 18 18:40:11.930574 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:11.930217    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zw2cl" event={"ID":"6455458b-83a0-4f35-857a-f11c3866299e","Type":"ContainerStarted","Data":"1419f65705963d97587417fc98932a92e33d62bb6ce3427fdff0559c08f9b5c6"}
Sep 18 18:40:11.930667 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:11.930639    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:40:11.948599 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:11.947657    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zw2cl" podStartSLOduration=33.675585274 podStartE2EDuration="35.947638511s" totalImagesPullingTime="2.272053237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:40:11.947218841 +0000 UTC m=+71.383797685" watchObservedRunningTime="2026-09-18 18:40:11.947638511 +0000 UTC m=+71.384217353"
Sep 18 18:40:12.198667 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:12.198643    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-rk7wk"]
Sep 18 18:40:12.201490 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:40:12.201466    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0801c387_848b_4c2f_8188_97b7720c46d8.slice/crio-b128b223d9255ed37e3d66c240a93bdc7804f8d8058fdd47b6210b7f8a7a0fb6 WatchSource:0}: Error finding container b128b223d9255ed37e3d66c240a93bdc7804f8d8058fdd47b6210b7f8a7a0fb6: Status 404 returned error can't find the container with id b128b223d9255ed37e3d66c240a93bdc7804f8d8058fdd47b6210b7f8a7a0fb6
Sep 18 18:40:12.934760 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:12.934721    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-rk7wk" event={"ID":"0801c387-848b-4c2f-8188-97b7720c46d8","Type":"ContainerStarted","Data":"b128b223d9255ed37e3d66c240a93bdc7804f8d8058fdd47b6210b7f8a7a0fb6"}
Sep 18 18:40:14.943456 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:14.943420    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w9bz8" event={"ID":"9ac6087e-296b-4917-bee6-8540193b8719","Type":"ContainerStarted","Data":"9ea55aa393961b7ce6aef2f6f79594ab1ef9d0884e4d830ef0bae3a610434ac0"}
Sep 18 18:40:14.957474 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:14.957420    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w9bz8" podStartSLOduration=67.218772594 podStartE2EDuration="1m11.957403784s" totalImagesPullingTime="4.73863119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:40:14.956004141 +0000 UTC m=+74.392582984" watchObservedRunningTime="2026-09-18 18:40:14.957403784 +0000 UTC m=+74.393982625"
Sep 18 18:40:15.025663 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.025635    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:40:15.025818 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.025675    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4lcv7\" (UID: \"f649578f-b8d4-4207-9dcc-69ce583fc6f6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"
Sep 18 18:40:15.025818 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.025719    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-p27wv\" (UID: \"9b2a9f30-7931-4049-8d50-64203df5c3fd\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:40:15.026459 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.026419    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7117c313-6fb4-48b0-8f3e-16e2c9c8d551-service-ca-bundle\") pod \"router-default-6bd44bd444-kdgsw\" (UID: \"7117c313-6fb4-48b0-8f3e-16e2c9c8d551\") " pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:40:15.028628 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.028601    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f649578f-b8d4-4207-9dcc-69ce583fc6f6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-4lcv7\" (UID: \"f649578f-b8d4-4207-9dcc-69ce583fc6f6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"
Sep 18 18:40:15.028749 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.028646    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2a9f30-7931-4049-8d50-64203df5c3fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-p27wv\" (UID: \"9b2a9f30-7931-4049-8d50-64203df5c3fd\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:40:15.191077 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.191045    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-n2h7g\""
Sep 18 18:40:15.200003 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.199937    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"
Sep 18 18:40:15.205277 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.205223    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-k7f25\""
Sep 18 18:40:15.211017 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.210995    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mq7c7\""
Sep 18 18:40:15.213889 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.213868    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:40:15.219657 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.219606    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"
Sep 18 18:40:15.484484 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.484409    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7"]
Sep 18 18:40:15.487457 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:40:15.487427    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf649578f_b8d4_4207_9dcc_69ce583fc6f6.slice/crio-b0b6d34400b39975b38a3dd6efe27ec810b0fb261551a93dcc703f55104a5e28 WatchSource:0}: Error finding container b0b6d34400b39975b38a3dd6efe27ec810b0fb261551a93dcc703f55104a5e28: Status 404 returned error can't find the container with id b0b6d34400b39975b38a3dd6efe27ec810b0fb261551a93dcc703f55104a5e28
Sep 18 18:40:15.517219 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.517055    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv"]
Sep 18 18:40:15.518789 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.518771    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6bd44bd444-kdgsw"]
Sep 18 18:40:15.519108 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:40:15.519083    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2a9f30_7931_4049_8d50_64203df5c3fd.slice/crio-ce000a910d760b77ebfc978b1b74c04598e06ed4d1913531f55cc8ff727a703e WatchSource:0}: Error finding container ce000a910d760b77ebfc978b1b74c04598e06ed4d1913531f55cc8ff727a703e: Status 404 returned error can't find the container with id ce000a910d760b77ebfc978b1b74c04598e06ed4d1913531f55cc8ff727a703e
Sep 18 18:40:15.521477 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:40:15.521397    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7117c313_6fb4_48b0_8f3e_16e2c9c8d551.slice/crio-ab96e68ef2311b0061c31f9515d7def3d4dd8b7204de33b2f444396eb4f76bd4 WatchSource:0}: Error finding container ab96e68ef2311b0061c31f9515d7def3d4dd8b7204de33b2f444396eb4f76bd4: Status 404 returned error can't find the container with id ab96e68ef2311b0061c31f9515d7def3d4dd8b7204de33b2f444396eb4f76bd4
Sep 18 18:40:15.949605 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.949565    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv" event={"ID":"9b2a9f30-7931-4049-8d50-64203df5c3fd","Type":"ContainerStarted","Data":"ce000a910d760b77ebfc978b1b74c04598e06ed4d1913531f55cc8ff727a703e"}
Sep 18 18:40:15.951013 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.950980    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7" event={"ID":"f649578f-b8d4-4207-9dcc-69ce583fc6f6","Type":"ContainerStarted","Data":"b0b6d34400b39975b38a3dd6efe27ec810b0fb261551a93dcc703f55104a5e28"}
Sep 18 18:40:15.952643 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.952617    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6bd44bd444-kdgsw" event={"ID":"7117c313-6fb4-48b0-8f3e-16e2c9c8d551","Type":"ContainerStarted","Data":"ce3dabc33604034ac7a80bfee7378b6017e3f2b45fc0a901e1caea82046715f4"}
Sep 18 18:40:15.952766 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.952664    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6bd44bd444-kdgsw" event={"ID":"7117c313-6fb4-48b0-8f3e-16e2c9c8d551","Type":"ContainerStarted","Data":"ab96e68ef2311b0061c31f9515d7def3d4dd8b7204de33b2f444396eb4f76bd4"}
Sep 18 18:40:15.973806 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:15.973753    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6bd44bd444-kdgsw" podStartSLOduration=32.973734015 podStartE2EDuration="32.973734015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:40:15.973177495 +0000 UTC m=+75.409756361" watchObservedRunningTime="2026-09-18 18:40:15.973734015 +0000 UTC m=+75.410312857"
Sep 18 18:40:16.214634 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:16.214554    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:40:16.217479 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:16.217456    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:40:16.958145 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:16.958110    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:40:16.959431 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:16.959408    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6bd44bd444-kdgsw"
Sep 18 18:40:17.966212 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:17.965705    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7" event={"ID":"f649578f-b8d4-4207-9dcc-69ce583fc6f6","Type":"ContainerStarted","Data":"bee8f47f2f91e6914a2e24f25645d74fbd98cf0b8e60b9ab5606e7c5e83c953f"}
Sep 18 18:40:17.967585 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:17.967526    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv" event={"ID":"9b2a9f30-7931-4049-8d50-64203df5c3fd","Type":"ContainerStarted","Data":"f13474281612d4fd6ab441d03dded0a4a05e7844abcd052ea95add379b1bec8a"}
Sep 18 18:40:17.999809 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:17.999757    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-4lcv7" podStartSLOduration=33.028189848 podStartE2EDuration="34.999737095s" totalImagesPullingTime="1.971547247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:40:17.998004639 +0000 UTC m=+77.434583477" watchObservedRunningTime="2026-09-18 18:40:17.999737095 +0000 UTC m=+77.436315938"
Sep 18 18:40:18.017176 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:18.017106    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-p27wv" podStartSLOduration=33.072553737 podStartE2EDuration="35.017091229s" totalImagesPullingTime="1.944537492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:40:18.016023899 +0000 UTC m=+77.452602739" watchObservedRunningTime="2026-09-18 18:40:18.017091229 +0000 UTC m=+77.453670073"
Sep 18 18:40:21.937731 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:21.937698    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zw2cl"
Sep 18 18:40:29.479968 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.479308    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ttrkh"]
Sep 18 18:40:29.485247 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.485223    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.487928 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.487774    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 18:40:29.488052 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.488031    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 18:40:29.488112 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.488066    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 18:40:29.488714 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.488521    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 18:40:29.488714 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.488625    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s6ph4\""
Sep 18 18:40:29.548668 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.548635    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b347122-1593-4a1a-b376-e0423c45ac44-sys\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.548668 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.548669    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-textfile\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.548905 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.548690    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b347122-1593-4a1a-b376-e0423c45ac44-root\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.548905 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.548714    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-tls\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.548905 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.548753    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7b4h\" (UniqueName: \"kubernetes.io/projected/1b347122-1593-4a1a-b376-e0423c45ac44-kube-api-access-s7b4h\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.548905 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.548779    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b347122-1593-4a1a-b376-e0423c45ac44-metrics-client-ca\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.548905 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.548838    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-accelerators-collector-config\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.549150 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.548917    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.549150 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.548946    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-wtmp\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.649699 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.649659    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b347122-1593-4a1a-b376-e0423c45ac44-metrics-client-ca\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.649883 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.649714    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-accelerators-collector-config\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.649883 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.649749    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.649883 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.649789    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-wtmp\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.649883 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.649851    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b347122-1593-4a1a-b376-e0423c45ac44-sys\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.649883 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.649873    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-textfile\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.650151 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.649896    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b347122-1593-4a1a-b376-e0423c45ac44-root\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.650151 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.649914    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-tls\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.650151 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.649929    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7b4h\" (UniqueName: \"kubernetes.io/projected/1b347122-1593-4a1a-b376-e0423c45ac44-kube-api-access-s7b4h\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.650410 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.650380    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b347122-1593-4a1a-b376-e0423c45ac44-metrics-client-ca\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.650553 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.650444    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b347122-1593-4a1a-b376-e0423c45ac44-sys\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.650553 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.650498    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b347122-1593-4a1a-b376-e0423c45ac44-root\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.650553 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.650380    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-accelerators-collector-config\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.650726 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.650529    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-wtmp\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.650726 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:40:29.650584    2558 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 18:40:29.650726 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:40:29.650638    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-tls podName:1b347122-1593-4a1a-b376-e0423c45ac44 nodeName:}" failed. No retries permitted until 2026-09-18 18:40:30.150620315 +0000 UTC m=+89.587199136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-tls") pod "node-exporter-ttrkh" (UID: "1b347122-1593-4a1a-b376-e0423c45ac44") : secret "node-exporter-tls" not found
Sep 18 18:40:29.650900 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.650740    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-textfile\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.653064 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.653043    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:29.661594 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:29.661570    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7b4h\" (UniqueName: \"kubernetes.io/projected/1b347122-1593-4a1a-b376-e0423c45ac44-kube-api-access-s7b4h\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:30.154465 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:30.154430    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-tls\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:30.156812 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:30.156786    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b347122-1593-4a1a-b376-e0423c45ac44-node-exporter-tls\") pod \"node-exporter-ttrkh\" (UID: \"1b347122-1593-4a1a-b376-e0423c45ac44\") " pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:30.399984 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:30.399921    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ttrkh"
Sep 18 18:40:30.414567 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:40:30.414525    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b347122_1593_4a1a_b376_e0423c45ac44.slice/crio-75628027eda9035b8297e2465d6596a4f2141268e01b0af7b49adbfbc8839e1a WatchSource:0}: Error finding container 75628027eda9035b8297e2465d6596a4f2141268e01b0af7b49adbfbc8839e1a: Status 404 returned error can't find the container with id 75628027eda9035b8297e2465d6596a4f2141268e01b0af7b49adbfbc8839e1a
Sep 18 18:40:30.931134 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:30.930248    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:40:31.043757 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:31.042670    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-rk7wk" event={"ID":"0801c387-848b-4c2f-8188-97b7720c46d8","Type":"ContainerStarted","Data":"72f89ba5698ba844808c543a12b5549e8451596b3686e3037d6b13c55b67ddf7"}
Sep 18 18:40:31.043757 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:31.043594    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-rk7wk"
Sep 18 18:40:31.046537 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:31.046464    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ttrkh" event={"ID":"1b347122-1593-4a1a-b376-e0423c45ac44","Type":"ContainerStarted","Data":"75628027eda9035b8297e2465d6596a4f2141268e01b0af7b49adbfbc8839e1a"}
Sep 18 18:40:31.054201 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:31.054145    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-rk7wk"
Sep 18 18:40:31.067844 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:31.067788    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-rk7wk" podStartSLOduration=2.036744175 podStartE2EDuration="20.067770063s" totalImagesPullingTime="18.031025888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:40:31.065099512 +0000 UTC m=+90.501678354" watchObservedRunningTime="2026-09-18 18:40:31.067770063 +0000 UTC m=+90.504348906"
Sep 18 18:40:33.058155 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:33.058114    2558 generic.go:356] "Generic (PLEG): container finished" podID="1b347122-1593-4a1a-b376-e0423c45ac44" containerID="17e605fd9536306116036445e9035cf2c339caa9036cb532147fa72f10a6b8c9" exitCode=0
Sep 18 18:40:33.058641 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:33.058220    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ttrkh" event={"ID":"1b347122-1593-4a1a-b376-e0423c45ac44","Type":"ContainerDied","Data":"17e605fd9536306116036445e9035cf2c339caa9036cb532147fa72f10a6b8c9"}
Sep 18 18:40:34.065401 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:34.065365    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ttrkh" event={"ID":"1b347122-1593-4a1a-b376-e0423c45ac44","Type":"ContainerStarted","Data":"46fc43c6f4c726692a16546e2f0d2fcf72161deec1bff1c5d681880a8ec6dcb2"}
Sep 18 18:40:34.065828 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:34.065425    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ttrkh" event={"ID":"1b347122-1593-4a1a-b376-e0423c45ac44","Type":"ContainerStarted","Data":"5deca1b617dd07fd2bb94e017df9d24ff7d0654db1805bceecd7bc31c4d06012"}
Sep 18 18:40:34.088675 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:34.088613    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ttrkh" podStartSLOduration=3.437388886 podStartE2EDuration="5.088594908s" totalImagesPullingTime="1.567995872s" totalInitContainerRuntime="83.21015ms" isStatefulPod=true podCreationTimestamp="2026-09-18 18:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:40:34.088332117 +0000 UTC m=+93.524910960" watchObservedRunningTime="2026-09-18 18:40:34.088594908 +0000 UTC m=+93.525173752"
Sep 18 18:40:34.946045 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:34.946008    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bdbd487b5-zx747"]
Sep 18 18:40:41.932844 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:41.932814    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xrssq"
Sep 18 18:40:59.974230 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:40:59.974165    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7bdbd487b5-zx747" podUID="92f09b3c-b43d-4de1-b12f-b1a1ae026f41" containerName="registry" containerID="cri-o://9943ec032b40983069cf9b4068faf4b4054073df1174111311fa0a63d806facd" gracePeriod=30
Sep 18 18:41:00.165450 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.165419    2558 generic.go:356] "Generic (PLEG): container finished" podID="92f09b3c-b43d-4de1-b12f-b1a1ae026f41" containerID="9943ec032b40983069cf9b4068faf4b4054073df1174111311fa0a63d806facd" exitCode=0
Sep 18 18:41:00.165582 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.165488    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bdbd487b5-zx747" event={"ID":"92f09b3c-b43d-4de1-b12f-b1a1ae026f41","Type":"ContainerDied","Data":"9943ec032b40983069cf9b4068faf4b4054073df1174111311fa0a63d806facd"}
Sep 18 18:41:00.243743 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.243680    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:41:00.411542 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.411507    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-bound-sa-token\") pod \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") "
Sep 18 18:41:00.411542 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.411545    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls\") pod \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") "
Sep 18 18:41:00.411743 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.411567    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-certificates\") pod \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") "
Sep 18 18:41:00.411784 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.411757    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-ca-trust-extracted\") pod \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") "
Sep 18 18:41:00.411822 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.411800    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-installation-pull-secrets\") pod \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") "
Sep 18 18:41:00.411868 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.411844    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-trusted-ca\") pod \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") "
Sep 18 18:41:00.411915 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.411882    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-image-registry-private-configuration\") pod \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") "
Sep 18 18:41:00.411969 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.411928    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r8gv\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-kube-api-access-9r8gv\") pod \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\" (UID: \"92f09b3c-b43d-4de1-b12f-b1a1ae026f41\") "
Sep 18 18:41:00.412026 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.411977    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "92f09b3c-b43d-4de1-b12f-b1a1ae026f41" (UID: "92f09b3c-b43d-4de1-b12f-b1a1ae026f41"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:41:00.412177 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.412153    2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-certificates\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:41:00.412516 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.412483    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "92f09b3c-b43d-4de1-b12f-b1a1ae026f41" (UID: "92f09b3c-b43d-4de1-b12f-b1a1ae026f41"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:41:00.414322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.414274    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-kube-api-access-9r8gv" (OuterVolumeSpecName: "kube-api-access-9r8gv") pod "92f09b3c-b43d-4de1-b12f-b1a1ae026f41" (UID: "92f09b3c-b43d-4de1-b12f-b1a1ae026f41"). InnerVolumeSpecName "kube-api-access-9r8gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:41:00.414437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.414407    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "92f09b3c-b43d-4de1-b12f-b1a1ae026f41" (UID: "92f09b3c-b43d-4de1-b12f-b1a1ae026f41"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:41:00.414510 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.414495    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "92f09b3c-b43d-4de1-b12f-b1a1ae026f41" (UID: "92f09b3c-b43d-4de1-b12f-b1a1ae026f41"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:41:00.414563 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.414477    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "92f09b3c-b43d-4de1-b12f-b1a1ae026f41" (UID: "92f09b3c-b43d-4de1-b12f-b1a1ae026f41"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:41:00.414604 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.414556    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "92f09b3c-b43d-4de1-b12f-b1a1ae026f41" (UID: "92f09b3c-b43d-4de1-b12f-b1a1ae026f41"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:41:00.420740 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.420711    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "92f09b3c-b43d-4de1-b12f-b1a1ae026f41" (UID: "92f09b3c-b43d-4de1-b12f-b1a1ae026f41"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:41:00.513396 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.513314    2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-bound-sa-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:41:00.513396 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.513341    2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-registry-tls\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:41:00.513396 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.513350    2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-ca-trust-extracted\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:41:00.513396 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.513360    2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-installation-pull-secrets\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:41:00.513396 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.513368    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-trusted-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:41:00.513396 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.513379    2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-image-registry-private-configuration\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:41:00.513396 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:00.513387    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r8gv\" (UniqueName: \"kubernetes.io/projected/92f09b3c-b43d-4de1-b12f-b1a1ae026f41-kube-api-access-9r8gv\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:41:01.170768 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:01.170728    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bdbd487b5-zx747" event={"ID":"92f09b3c-b43d-4de1-b12f-b1a1ae026f41","Type":"ContainerDied","Data":"1f6281c549a46cba6fb35c29a8ede46f155f13f344389ce84dd6fbe1bc9306a7"}
Sep 18 18:41:01.170768 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:01.170771    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bdbd487b5-zx747"
Sep 18 18:41:01.171249 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:01.170841    2558 scope.go:117] "RemoveContainer" containerID="9943ec032b40983069cf9b4068faf4b4054073df1174111311fa0a63d806facd"
Sep 18 18:41:01.187808 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:01.187782    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bdbd487b5-zx747"]
Sep 18 18:41:01.191975 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:01.191954    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7bdbd487b5-zx747"]
Sep 18 18:41:02.573490 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:02.573456    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92f09b3c-b43d-4de1-b12f-b1a1ae026f41" path="/var/lib/kubelet/pods/92f09b3c-b43d-4de1-b12f-b1a1ae026f41/volumes"
Sep 18 18:41:15.226241 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:15.226203    2558 generic.go:356] "Generic (PLEG): container finished" podID="63302165-e325-4efd-88d8-dda368f6066e" containerID="25e250bb4ff7d4377d901d5384f8b21246d74395fce9423d314a14d0f43fea51" exitCode=0
Sep 18 18:41:15.226241 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:15.226217    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-q64fh" event={"ID":"63302165-e325-4efd-88d8-dda368f6066e","Type":"ContainerDied","Data":"25e250bb4ff7d4377d901d5384f8b21246d74395fce9423d314a14d0f43fea51"}
Sep 18 18:41:15.226832 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:15.226658    2558 scope.go:117] "RemoveContainer" containerID="25e250bb4ff7d4377d901d5384f8b21246d74395fce9423d314a14d0f43fea51"
Sep 18 18:41:15.227849 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:15.227829    2558 generic.go:356] "Generic (PLEG): container finished" podID="8b41f87c-1c13-44ea-ae32-8ff22711e45e" containerID="9e2111e503da03a59744f086ab0213f661796373172f2e14ee1395487ba04215" exitCode=0
Sep 18 18:41:15.227946 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:15.227872    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj" event={"ID":"8b41f87c-1c13-44ea-ae32-8ff22711e45e","Type":"ContainerDied","Data":"9e2111e503da03a59744f086ab0213f661796373172f2e14ee1395487ba04215"}
Sep 18 18:41:15.228178 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:15.228164    2558 scope.go:117] "RemoveContainer" containerID="9e2111e503da03a59744f086ab0213f661796373172f2e14ee1395487ba04215"
Sep 18 18:41:16.233662 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:16.233629    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-n7tgj" event={"ID":"8b41f87c-1c13-44ea-ae32-8ff22711e45e","Type":"ContainerStarted","Data":"d157f5d170e9341d907075fe4e8054190a03313758dcaa216400c7d872fc9588"}
Sep 18 18:41:16.235744 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:16.235713    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-q64fh" event={"ID":"63302165-e325-4efd-88d8-dda368f6066e","Type":"ContainerStarted","Data":"cdc5126d1be39f0d5ba08629e8ef32327310272f8a7bd5c36eb207af4111ca20"}
Sep 18 18:41:20.251776 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:20.251739    2558 generic.go:356] "Generic (PLEG): container finished" podID="cccc2e3c-0d90-42aa-a136-069631e71567" containerID="801db0f4cd15ce9a50bd35785504c2b2dbf217198965262595014555943ccbd3" exitCode=0
Sep 18 18:41:20.251776 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:20.251778    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr" event={"ID":"cccc2e3c-0d90-42aa-a136-069631e71567","Type":"ContainerDied","Data":"801db0f4cd15ce9a50bd35785504c2b2dbf217198965262595014555943ccbd3"}
Sep 18 18:41:20.252238 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:20.252086    2558 scope.go:117] "RemoveContainer" containerID="801db0f4cd15ce9a50bd35785504c2b2dbf217198965262595014555943ccbd3"
Sep 18 18:41:21.257612 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:21.257574    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-84wlr" event={"ID":"cccc2e3c-0d90-42aa-a136-069631e71567","Type":"ContainerStarted","Data":"68b0a5b7aa7f24601729633a99425b17e9332b76f2589d4642d6175e3df239e1"}
Sep 18 18:41:53.758437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.758356    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6d44ccc684-h79gv"]
Sep 18 18:41:53.758876 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.758642    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f09b3c-b43d-4de1-b12f-b1a1ae026f41" containerName="registry"
Sep 18 18:41:53.758876 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.758653    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f09b3c-b43d-4de1-b12f-b1a1ae026f41" containerName="registry"
Sep 18 18:41:53.758876 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.758706    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="92f09b3c-b43d-4de1-b12f-b1a1ae026f41" containerName="registry"
Sep 18 18:41:53.761526 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.761504    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.764223 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.764166    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 18:41:53.764223 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.764166    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 18:41:53.764442 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.764257    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-826lz\""
Sep 18 18:41:53.764545 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.764475    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 18:41:53.764659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.764555    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 18:41:53.765201 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.765174    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 18:41:53.777301 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.777278    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 18:41:53.778502 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.778476    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d44ccc684-h79gv"]
Sep 18 18:41:53.801943 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.801913    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2b79b04-761c-4dc3-829e-a3a779aac406-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.802063 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.801957    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f2b79b04-761c-4dc3-829e-a3a779aac406-telemeter-client-tls\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.802063 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.801985    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f2b79b04-761c-4dc3-829e-a3a779aac406-secret-telemeter-client\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.802063 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.802007    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctvxk\" (UniqueName: \"kubernetes.io/projected/f2b79b04-761c-4dc3-829e-a3a779aac406-kube-api-access-ctvxk\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.802063 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.802031    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2b79b04-761c-4dc3-829e-a3a779aac406-serving-certs-ca-bundle\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.802063 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.802047    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2b79b04-761c-4dc3-829e-a3a779aac406-metrics-client-ca\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.802278 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.802079    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f2b79b04-761c-4dc3-829e-a3a779aac406-federate-client-tls\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.802278 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.802158    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2b79b04-761c-4dc3-829e-a3a779aac406-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.902728 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.902696    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f2b79b04-761c-4dc3-829e-a3a779aac406-secret-telemeter-client\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.902728 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.902731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctvxk\" (UniqueName: \"kubernetes.io/projected/f2b79b04-761c-4dc3-829e-a3a779aac406-kube-api-access-ctvxk\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.902964 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.902760    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2b79b04-761c-4dc3-829e-a3a779aac406-serving-certs-ca-bundle\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.902964 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.902914    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2b79b04-761c-4dc3-829e-a3a779aac406-metrics-client-ca\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.903076 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.902997    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f2b79b04-761c-4dc3-829e-a3a779aac406-federate-client-tls\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.903129 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.903083    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2b79b04-761c-4dc3-829e-a3a779aac406-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.903181 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.903142    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2b79b04-761c-4dc3-829e-a3a779aac406-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.903291 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.903268    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f2b79b04-761c-4dc3-829e-a3a779aac406-telemeter-client-tls\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.903586 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.903561    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2b79b04-761c-4dc3-829e-a3a779aac406-serving-certs-ca-bundle\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.903848 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.903816    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2b79b04-761c-4dc3-829e-a3a779aac406-metrics-client-ca\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.904348 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.904327    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2b79b04-761c-4dc3-829e-a3a779aac406-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.906015 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.905992    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f2b79b04-761c-4dc3-829e-a3a779aac406-secret-telemeter-client\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.906128 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.906081    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2b79b04-761c-4dc3-829e-a3a779aac406-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.906218 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.906166    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f2b79b04-761c-4dc3-829e-a3a779aac406-telemeter-client-tls\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.906309 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.906291    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f2b79b04-761c-4dc3-829e-a3a779aac406-federate-client-tls\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:53.911597 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:53.911575    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctvxk\" (UniqueName: \"kubernetes.io/projected/f2b79b04-761c-4dc3-829e-a3a779aac406-kube-api-access-ctvxk\") pod \"telemeter-client-6d44ccc684-h79gv\" (UID: \"f2b79b04-761c-4dc3-829e-a3a779aac406\") " pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:54.072414 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:54.072330    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv"
Sep 18 18:41:54.348928 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:54.348903    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d44ccc684-h79gv"]
Sep 18 18:41:54.350700 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:41:54.350671    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b79b04_761c_4dc3_829e_a3a779aac406.slice/crio-491b93b5c6ac60af279f63b7504c1c682b266c3c3edc3b2f0b3916ed8165cdd2 WatchSource:0}: Error finding container 491b93b5c6ac60af279f63b7504c1c682b266c3c3edc3b2f0b3916ed8165cdd2: Status 404 returned error can't find the container with id 491b93b5c6ac60af279f63b7504c1c682b266c3c3edc3b2f0b3916ed8165cdd2
Sep 18 18:41:54.390810 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:54.390768    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv" event={"ID":"f2b79b04-761c-4dc3-829e-a3a779aac406","Type":"ContainerStarted","Data":"491b93b5c6ac60af279f63b7504c1c682b266c3c3edc3b2f0b3916ed8165cdd2"}
Sep 18 18:41:57.405377 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:57.405332    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv" event={"ID":"f2b79b04-761c-4dc3-829e-a3a779aac406","Type":"ContainerStarted","Data":"a2a0f00e0b4ec3516cb6527261118dae9bb1f1cd8d4a8134d4856f7382aac401"}
Sep 18 18:41:58.411751 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:58.411706    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv" event={"ID":"f2b79b04-761c-4dc3-829e-a3a779aac406","Type":"ContainerStarted","Data":"659574f64ea0c3164bfe0ce971c2c6709dc1e1503a0192fb900099820b280207"}
Sep 18 18:41:58.412203 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:58.411776    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv" event={"ID":"f2b79b04-761c-4dc3-829e-a3a779aac406","Type":"ContainerStarted","Data":"8a2dcdf85592321f99a96fdaab227cdd2899dcbcb11803a6d3aab0da30efa8f4"}
Sep 18 18:41:58.437413 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:41:58.437351    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6d44ccc684-h79gv" podStartSLOduration=2.350653312 podStartE2EDuration="5.437332741s" totalImagesPullingTime="3.086679429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:41:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:41:58.435256701 +0000 UTC m=+177.871835544" watchObservedRunningTime="2026-09-18 18:41:58.437332741 +0000 UTC m=+177.873911582"
Sep 18 18:44:02.514489 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:02.514461    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 18:44:02.517149 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:02.517128    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 18:44:02.519778 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:02.519758    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 18:44:02.522893 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:02.522876    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 18:44:02.525229 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:02.525212    2558 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 18:44:11.539336 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.539295    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"]
Sep 18 18:44:11.542486 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.542471    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"
Sep 18 18:44:11.544803 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.544780    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 18:44:11.545457 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.545437    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 18:44:11.545543 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.545437    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ktfvw\""
Sep 18 18:44:11.551637 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.551617    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"]
Sep 18 18:44:11.592536 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.592505    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vxq9\" (UniqueName: \"kubernetes.io/projected/b9bf46eb-fcfb-4979-a14a-6f03b9962060-kube-api-access-4vxq9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd\" (UID: \"b9bf46eb-fcfb-4979-a14a-6f03b9962060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"
Sep 18 18:44:11.592669 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.592551    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9bf46eb-fcfb-4979-a14a-6f03b9962060-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd\" (UID: \"b9bf46eb-fcfb-4979-a14a-6f03b9962060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"
Sep 18 18:44:11.592669 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.592614    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9bf46eb-fcfb-4979-a14a-6f03b9962060-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd\" (UID: \"b9bf46eb-fcfb-4979-a14a-6f03b9962060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"
Sep 18 18:44:11.694066 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.694025    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vxq9\" (UniqueName: \"kubernetes.io/projected/b9bf46eb-fcfb-4979-a14a-6f03b9962060-kube-api-access-4vxq9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd\" (UID: \"b9bf46eb-fcfb-4979-a14a-6f03b9962060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"
Sep 18 18:44:11.694277 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.694125    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9bf46eb-fcfb-4979-a14a-6f03b9962060-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd\" (UID: \"b9bf46eb-fcfb-4979-a14a-6f03b9962060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"
Sep 18 18:44:11.694277 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.694170    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9bf46eb-fcfb-4979-a14a-6f03b9962060-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd\" (UID: \"b9bf46eb-fcfb-4979-a14a-6f03b9962060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"
Sep 18 18:44:11.694609 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.694588    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9bf46eb-fcfb-4979-a14a-6f03b9962060-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd\" (UID: \"b9bf46eb-fcfb-4979-a14a-6f03b9962060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"
Sep 18 18:44:11.694688 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.694609    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9bf46eb-fcfb-4979-a14a-6f03b9962060-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd\" (UID: \"b9bf46eb-fcfb-4979-a14a-6f03b9962060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"
Sep 18 18:44:11.702268 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.702246    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vxq9\" (UniqueName: \"kubernetes.io/projected/b9bf46eb-fcfb-4979-a14a-6f03b9962060-kube-api-access-4vxq9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd\" (UID: \"b9bf46eb-fcfb-4979-a14a-6f03b9962060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"
Sep 18 18:44:11.850850 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:11.850816    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"
Sep 18 18:44:12.121470 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:12.121444    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"]
Sep 18 18:44:12.123497 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:44:12.123474    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bf46eb_fcfb_4979_a14a_6f03b9962060.slice/crio-1a7ec4e2289e4e3300aa79b98b56ef696c9aa772ed2bb2a6b542313313bec04b WatchSource:0}: Error finding container 1a7ec4e2289e4e3300aa79b98b56ef696c9aa772ed2bb2a6b542313313bec04b: Status 404 returned error can't find the container with id 1a7ec4e2289e4e3300aa79b98b56ef696c9aa772ed2bb2a6b542313313bec04b
Sep 18 18:44:12.125736 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:12.125716    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:44:12.910535 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:12.910502    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd" event={"ID":"b9bf46eb-fcfb-4979-a14a-6f03b9962060","Type":"ContainerStarted","Data":"1a7ec4e2289e4e3300aa79b98b56ef696c9aa772ed2bb2a6b542313313bec04b"}
Sep 18 18:44:18.938612 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:18.938568    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd" event={"ID":"b9bf46eb-fcfb-4979-a14a-6f03b9962060","Type":"ContainerStarted","Data":"2f1969ba836ace11976480c6658b8bd9d399ef06ab2017d76baf81426e327716"}
Sep 18 18:44:19.943729 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:19.943691    2558 generic.go:356] "Generic (PLEG): container finished" podID="b9bf46eb-fcfb-4979-a14a-6f03b9962060" containerID="2f1969ba836ace11976480c6658b8bd9d399ef06ab2017d76baf81426e327716" exitCode=0
Sep 18 18:44:19.944172 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:19.943781    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd" event={"ID":"b9bf46eb-fcfb-4979-a14a-6f03b9962060","Type":"ContainerDied","Data":"2f1969ba836ace11976480c6658b8bd9d399ef06ab2017d76baf81426e327716"}
Sep 18 18:44:21.954335 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:21.954303    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd" event={"ID":"b9bf46eb-fcfb-4979-a14a-6f03b9962060","Type":"ContainerStarted","Data":"8f8ebf17b5c91e15f66fa6351bd5803336676b2b91f0a823fee5222d59891d9d"}
Sep 18 18:44:22.959875 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:22.959839    2558 generic.go:356] "Generic (PLEG): container finished" podID="b9bf46eb-fcfb-4979-a14a-6f03b9962060" containerID="8f8ebf17b5c91e15f66fa6351bd5803336676b2b91f0a823fee5222d59891d9d" exitCode=0
Sep 18 18:44:22.960289 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:22.959928    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd" event={"ID":"b9bf46eb-fcfb-4979-a14a-6f03b9962060","Type":"ContainerDied","Data":"8f8ebf17b5c91e15f66fa6351bd5803336676b2b91f0a823fee5222d59891d9d"}
Sep 18 18:44:30.995625 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:30.995583    2558 generic.go:356] "Generic (PLEG): container finished" podID="b9bf46eb-fcfb-4979-a14a-6f03b9962060" containerID="48edb492ea3e98433b17edd47aeba94e293ea44298f53b17a7d23e4aa6aef9df" exitCode=0
Sep 18 18:44:30.995977 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:30.995662    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd" event={"ID":"b9bf46eb-fcfb-4979-a14a-6f03b9962060","Type":"ContainerDied","Data":"48edb492ea3e98433b17edd47aeba94e293ea44298f53b17a7d23e4aa6aef9df"}
Sep 18 18:44:32.121999 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:32.121975    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"
Sep 18 18:44:32.268815 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:32.268731    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9bf46eb-fcfb-4979-a14a-6f03b9962060-util\") pod \"b9bf46eb-fcfb-4979-a14a-6f03b9962060\" (UID: \"b9bf46eb-fcfb-4979-a14a-6f03b9962060\") "
Sep 18 18:44:32.268815 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:32.268770    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9bf46eb-fcfb-4979-a14a-6f03b9962060-bundle\") pod \"b9bf46eb-fcfb-4979-a14a-6f03b9962060\" (UID: \"b9bf46eb-fcfb-4979-a14a-6f03b9962060\") "
Sep 18 18:44:32.268815 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:32.268816    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vxq9\" (UniqueName: \"kubernetes.io/projected/b9bf46eb-fcfb-4979-a14a-6f03b9962060-kube-api-access-4vxq9\") pod \"b9bf46eb-fcfb-4979-a14a-6f03b9962060\" (UID: \"b9bf46eb-fcfb-4979-a14a-6f03b9962060\") "
Sep 18 18:44:32.269618 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:32.269587    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bf46eb-fcfb-4979-a14a-6f03b9962060-bundle" (OuterVolumeSpecName: "bundle") pod "b9bf46eb-fcfb-4979-a14a-6f03b9962060" (UID: "b9bf46eb-fcfb-4979-a14a-6f03b9962060"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:44:32.271012 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:32.270982    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bf46eb-fcfb-4979-a14a-6f03b9962060-kube-api-access-4vxq9" (OuterVolumeSpecName: "kube-api-access-4vxq9") pod "b9bf46eb-fcfb-4979-a14a-6f03b9962060" (UID: "b9bf46eb-fcfb-4979-a14a-6f03b9962060"). InnerVolumeSpecName "kube-api-access-4vxq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:44:32.273350 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:32.273329    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bf46eb-fcfb-4979-a14a-6f03b9962060-util" (OuterVolumeSpecName: "util") pod "b9bf46eb-fcfb-4979-a14a-6f03b9962060" (UID: "b9bf46eb-fcfb-4979-a14a-6f03b9962060"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:44:32.369892 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:32.369862    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9bf46eb-fcfb-4979-a14a-6f03b9962060-bundle\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:44:32.369892 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:32.369888    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vxq9\" (UniqueName: \"kubernetes.io/projected/b9bf46eb-fcfb-4979-a14a-6f03b9962060-kube-api-access-4vxq9\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:44:32.369892 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:32.369898    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9bf46eb-fcfb-4979-a14a-6f03b9962060-util\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:44:33.004864 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:33.004824    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd" event={"ID":"b9bf46eb-fcfb-4979-a14a-6f03b9962060","Type":"ContainerDied","Data":"1a7ec4e2289e4e3300aa79b98b56ef696c9aa772ed2bb2a6b542313313bec04b"}
Sep 18 18:44:33.005031 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:33.004882    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a7ec4e2289e4e3300aa79b98b56ef696c9aa772ed2bb2a6b542313313bec04b"
Sep 18 18:44:33.005031 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:33.004845    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407xvmd"
Sep 18 18:44:39.759763 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:39.759722    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wt8ph"]
Sep 18 18:44:39.760150 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:39.759995    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9bf46eb-fcfb-4979-a14a-6f03b9962060" containerName="util"
Sep 18 18:44:39.760150 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:39.760006    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9bf46eb-fcfb-4979-a14a-6f03b9962060" containerName="util"
Sep 18 18:44:39.760150 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:39.760018    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9bf46eb-fcfb-4979-a14a-6f03b9962060" containerName="extract"
Sep 18 18:44:39.760150 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:39.760025    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9bf46eb-fcfb-4979-a14a-6f03b9962060" containerName="extract"
Sep 18 18:44:39.760150 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:39.760033    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9bf46eb-fcfb-4979-a14a-6f03b9962060" containerName="pull"
Sep 18 18:44:39.760150 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:39.760039    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9bf46eb-fcfb-4979-a14a-6f03b9962060" containerName="pull"
Sep 18 18:44:39.760150 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:39.760094    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9bf46eb-fcfb-4979-a14a-6f03b9962060" containerName="extract"
Sep 18 18:44:39.766801 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:39.766784    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wt8ph"
Sep 18 18:44:39.769417 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:39.769395    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-f5k9z\""
Sep 18 18:44:39.769536 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:39.769468    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 18 18:44:39.769748 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:39.769734    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 18 18:44:39.775082 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:39.775064    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wt8ph"]
Sep 18 18:44:39.930063 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:39.930016    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrsnm\" (UniqueName: \"kubernetes.io/projected/0d708a3f-0cbe-4661-979d-c3c45ba54a51-kube-api-access-lrsnm\") pod \"cert-manager-operator-controller-manager-55c444bf86-wt8ph\" (UID: \"0d708a3f-0cbe-4661-979d-c3c45ba54a51\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wt8ph"
Sep 18 18:44:39.930264 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:39.930083    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d708a3f-0cbe-4661-979d-c3c45ba54a51-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wt8ph\" (UID: \"0d708a3f-0cbe-4661-979d-c3c45ba54a51\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wt8ph"
Sep 18 18:44:40.030511 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:40.030433    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrsnm\" (UniqueName: \"kubernetes.io/projected/0d708a3f-0cbe-4661-979d-c3c45ba54a51-kube-api-access-lrsnm\") pod \"cert-manager-operator-controller-manager-55c444bf86-wt8ph\" (UID: \"0d708a3f-0cbe-4661-979d-c3c45ba54a51\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wt8ph"
Sep 18 18:44:40.030511 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:40.030487    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d708a3f-0cbe-4661-979d-c3c45ba54a51-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wt8ph\" (UID: \"0d708a3f-0cbe-4661-979d-c3c45ba54a51\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wt8ph"
Sep 18 18:44:40.030832 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:40.030816    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d708a3f-0cbe-4661-979d-c3c45ba54a51-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wt8ph\" (UID: \"0d708a3f-0cbe-4661-979d-c3c45ba54a51\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wt8ph"
Sep 18 18:44:40.041854 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:40.041830    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrsnm\" (UniqueName: \"kubernetes.io/projected/0d708a3f-0cbe-4661-979d-c3c45ba54a51-kube-api-access-lrsnm\") pod \"cert-manager-operator-controller-manager-55c444bf86-wt8ph\" (UID: \"0d708a3f-0cbe-4661-979d-c3c45ba54a51\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wt8ph"
Sep 18 18:44:40.076689 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:40.076664    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wt8ph"
Sep 18 18:44:40.354275 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:40.354248    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wt8ph"]
Sep 18 18:44:40.357037 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:44:40.357013    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d708a3f_0cbe_4661_979d_c3c45ba54a51.slice/crio-74d43a20e0a8f9186399e1af6a0e3017995690e8d537db5e35133e9d013f117d WatchSource:0}: Error finding container 74d43a20e0a8f9186399e1af6a0e3017995690e8d537db5e35133e9d013f117d: Status 404 returned error can't find the container with id 74d43a20e0a8f9186399e1af6a0e3017995690e8d537db5e35133e9d013f117d
Sep 18 18:44:41.038385 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:41.038299    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wt8ph" event={"ID":"0d708a3f-0cbe-4661-979d-c3c45ba54a51","Type":"ContainerStarted","Data":"74d43a20e0a8f9186399e1af6a0e3017995690e8d537db5e35133e9d013f117d"}
Sep 18 18:44:43.049751 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:43.049718    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wt8ph" event={"ID":"0d708a3f-0cbe-4661-979d-c3c45ba54a51","Type":"ContainerStarted","Data":"0f4c0da56a719edde84869c682e1096d4bee71d0e0a175b4f9b46358b315d5b8"}
Sep 18 18:44:43.073138 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:43.073086    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wt8ph" podStartSLOduration=2.007596088 podStartE2EDuration="4.073070033s" totalImagesPullingTime="2.065473945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:44:43.072314551 +0000 UTC m=+342.508893392" watchObservedRunningTime="2026-09-18 18:44:43.073070033 +0000 UTC m=+342.509648875"
Sep 18 18:44:44.235071 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.235036    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"]
Sep 18 18:44:44.238573 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.238557    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"
Sep 18 18:44:44.241075 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.241050    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ktfvw\""
Sep 18 18:44:44.241239 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.241111    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 18:44:44.241736 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.241721    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 18:44:44.251838 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.251817    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"]
Sep 18 18:44:44.261535 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.261513    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkrx8\" (UniqueName: \"kubernetes.io/projected/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-kube-api-access-zkrx8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42\" (UID: \"2cae6749-1b21-456d-b7d3-e43fe02a9ec8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"
Sep 18 18:44:44.261626 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.261555    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42\" (UID: \"2cae6749-1b21-456d-b7d3-e43fe02a9ec8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"
Sep 18 18:44:44.261626 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.261577    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42\" (UID: \"2cae6749-1b21-456d-b7d3-e43fe02a9ec8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"
Sep 18 18:44:44.362749 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.362707    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrx8\" (UniqueName: \"kubernetes.io/projected/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-kube-api-access-zkrx8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42\" (UID: \"2cae6749-1b21-456d-b7d3-e43fe02a9ec8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"
Sep 18 18:44:44.362914 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.362765    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42\" (UID: \"2cae6749-1b21-456d-b7d3-e43fe02a9ec8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"
Sep 18 18:44:44.362914 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.362794    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42\" (UID: \"2cae6749-1b21-456d-b7d3-e43fe02a9ec8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"
Sep 18 18:44:44.363172 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.363150    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42\" (UID: \"2cae6749-1b21-456d-b7d3-e43fe02a9ec8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"
Sep 18 18:44:44.363274 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.363179    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42\" (UID: \"2cae6749-1b21-456d-b7d3-e43fe02a9ec8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"
Sep 18 18:44:44.370818 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.370797    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkrx8\" (UniqueName: \"kubernetes.io/projected/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-kube-api-access-zkrx8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42\" (UID: \"2cae6749-1b21-456d-b7d3-e43fe02a9ec8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"
Sep 18 18:44:44.548110 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.548033    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"
Sep 18 18:44:44.817313 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:44.817228    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"]
Sep 18 18:44:44.819906 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:44:44.819871    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cae6749_1b21_456d_b7d3_e43fe02a9ec8.slice/crio-cca7eea714b9c4c8dcd90f943e76020f560ec209ecff671452fe0207ad6646dd WatchSource:0}: Error finding container cca7eea714b9c4c8dcd90f943e76020f560ec209ecff671452fe0207ad6646dd: Status 404 returned error can't find the container with id cca7eea714b9c4c8dcd90f943e76020f560ec209ecff671452fe0207ad6646dd
Sep 18 18:44:45.059600 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:45.059571    2558 generic.go:356] "Generic (PLEG): container finished" podID="2cae6749-1b21-456d-b7d3-e43fe02a9ec8" containerID="0db43187e95be6c90edd1c594b255703d472c9f1d45c4b62c2bbab34bb8e5eae" exitCode=0
Sep 18 18:44:45.059767 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:45.059630    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42" event={"ID":"2cae6749-1b21-456d-b7d3-e43fe02a9ec8","Type":"ContainerDied","Data":"0db43187e95be6c90edd1c594b255703d472c9f1d45c4b62c2bbab34bb8e5eae"}
Sep 18 18:44:45.059767 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:45.059669    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42" event={"ID":"2cae6749-1b21-456d-b7d3-e43fe02a9ec8","Type":"ContainerStarted","Data":"cca7eea714b9c4c8dcd90f943e76020f560ec209ecff671452fe0207ad6646dd"}
Sep 18 18:44:45.963562 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:45.963529    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-d5b8d"]
Sep 18 18:44:45.966603 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:45.966588    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-d5b8d"
Sep 18 18:44:45.970179 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:45.970157    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 18:44:45.970863 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:45.970839    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-bc778\""
Sep 18 18:44:45.971028 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:45.971010    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 18:44:45.974601 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:45.974581    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/812cc9ce-499b-4e88-9040-7b827122121f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-d5b8d\" (UID: \"812cc9ce-499b-4e88-9040-7b827122121f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-d5b8d"
Sep 18 18:44:45.974693 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:45.974634    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9ntw\" (UniqueName: \"kubernetes.io/projected/812cc9ce-499b-4e88-9040-7b827122121f-kube-api-access-n9ntw\") pod \"cert-manager-cainjector-559cdd8c87-d5b8d\" (UID: \"812cc9ce-499b-4e88-9040-7b827122121f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-d5b8d"
Sep 18 18:44:45.994281 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:45.994258    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-d5b8d"]
Sep 18 18:44:46.075763 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:46.075733    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/812cc9ce-499b-4e88-9040-7b827122121f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-d5b8d\" (UID: \"812cc9ce-499b-4e88-9040-7b827122121f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-d5b8d"
Sep 18 18:44:46.075921 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:46.075787    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9ntw\" (UniqueName: \"kubernetes.io/projected/812cc9ce-499b-4e88-9040-7b827122121f-kube-api-access-n9ntw\") pod \"cert-manager-cainjector-559cdd8c87-d5b8d\" (UID: \"812cc9ce-499b-4e88-9040-7b827122121f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-d5b8d"
Sep 18 18:44:46.084540 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:46.084507    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/812cc9ce-499b-4e88-9040-7b827122121f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-d5b8d\" (UID: \"812cc9ce-499b-4e88-9040-7b827122121f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-d5b8d"
Sep 18 18:44:46.084649 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:46.084549    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9ntw\" (UniqueName: \"kubernetes.io/projected/812cc9ce-499b-4e88-9040-7b827122121f-kube-api-access-n9ntw\") pod \"cert-manager-cainjector-559cdd8c87-d5b8d\" (UID: \"812cc9ce-499b-4e88-9040-7b827122121f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-d5b8d"
Sep 18 18:44:46.284853 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:46.284770    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-d5b8d"
Sep 18 18:44:46.556125 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:46.556096    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-d5b8d"]
Sep 18 18:44:46.558589 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:44:46.558561    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod812cc9ce_499b_4e88_9040_7b827122121f.slice/crio-95e208ed8b6d99b00702c59334522922a0ef663736f96f62abbcf18da8f82a2f WatchSource:0}: Error finding container 95e208ed8b6d99b00702c59334522922a0ef663736f96f62abbcf18da8f82a2f: Status 404 returned error can't find the container with id 95e208ed8b6d99b00702c59334522922a0ef663736f96f62abbcf18da8f82a2f
Sep 18 18:44:47.070702 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:47.070663    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-d5b8d" event={"ID":"812cc9ce-499b-4e88-9040-7b827122121f","Type":"ContainerStarted","Data":"95e208ed8b6d99b00702c59334522922a0ef663736f96f62abbcf18da8f82a2f"}
Sep 18 18:44:49.083508 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:49.083471    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-d5b8d" event={"ID":"812cc9ce-499b-4e88-9040-7b827122121f","Type":"ContainerStarted","Data":"827de2ad0e75357e173314de34d452027de8f0e6da1349b8bb8439be83674b18"}
Sep 18 18:44:49.100293 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:49.100236    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-d5b8d" podStartSLOduration=1.666988457 podStartE2EDuration="4.100214664s" totalImagesPullingTime="2.433226207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:44:49.098968581 +0000 UTC m=+348.535547424" watchObservedRunningTime="2026-09-18 18:44:49.100214664 +0000 UTC m=+348.536793508"
Sep 18 18:44:51.095978 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:51.095941    2558 generic.go:356] "Generic (PLEG): container finished" podID="2cae6749-1b21-456d-b7d3-e43fe02a9ec8" containerID="8e1b53520eaafbc63199ae6b24cedd1fb120024973c38053761c92c16b88d96f" exitCode=0
Sep 18 18:44:51.096404 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:51.095983    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42" event={"ID":"2cae6749-1b21-456d-b7d3-e43fe02a9ec8","Type":"ContainerDied","Data":"8e1b53520eaafbc63199ae6b24cedd1fb120024973c38053761c92c16b88d96f"}
Sep 18 18:44:52.102518 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:52.102486    2558 generic.go:356] "Generic (PLEG): container finished" podID="2cae6749-1b21-456d-b7d3-e43fe02a9ec8" containerID="3fba7d23e157aa1d20b9caa1008616855aaef29db8435f7b74a88d4b102f7827" exitCode=0
Sep 18 18:44:52.102900 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:52.102528    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42" event={"ID":"2cae6749-1b21-456d-b7d3-e43fe02a9ec8","Type":"ContainerDied","Data":"3fba7d23e157aa1d20b9caa1008616855aaef29db8435f7b74a88d4b102f7827"}
Sep 18 18:44:53.229866 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:53.229844    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"
Sep 18 18:44:53.334682 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:53.334651    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-bundle\") pod \"2cae6749-1b21-456d-b7d3-e43fe02a9ec8\" (UID: \"2cae6749-1b21-456d-b7d3-e43fe02a9ec8\") "
Sep 18 18:44:53.334871 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:53.334696    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-util\") pod \"2cae6749-1b21-456d-b7d3-e43fe02a9ec8\" (UID: \"2cae6749-1b21-456d-b7d3-e43fe02a9ec8\") "
Sep 18 18:44:53.334871 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:53.334716    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkrx8\" (UniqueName: \"kubernetes.io/projected/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-kube-api-access-zkrx8\") pod \"2cae6749-1b21-456d-b7d3-e43fe02a9ec8\" (UID: \"2cae6749-1b21-456d-b7d3-e43fe02a9ec8\") "
Sep 18 18:44:53.335065 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:53.335031    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-bundle" (OuterVolumeSpecName: "bundle") pod "2cae6749-1b21-456d-b7d3-e43fe02a9ec8" (UID: "2cae6749-1b21-456d-b7d3-e43fe02a9ec8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:44:53.336975 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:53.336947    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-kube-api-access-zkrx8" (OuterVolumeSpecName: "kube-api-access-zkrx8") pod "2cae6749-1b21-456d-b7d3-e43fe02a9ec8" (UID: "2cae6749-1b21-456d-b7d3-e43fe02a9ec8"). InnerVolumeSpecName "kube-api-access-zkrx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:44:53.339481 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:53.339455    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-util" (OuterVolumeSpecName: "util") pod "2cae6749-1b21-456d-b7d3-e43fe02a9ec8" (UID: "2cae6749-1b21-456d-b7d3-e43fe02a9ec8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:44:53.435936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:53.435850    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-util\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:44:53.435936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:53.435886    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkrx8\" (UniqueName: \"kubernetes.io/projected/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-kube-api-access-zkrx8\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:44:53.435936 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:53.435899    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cae6749-1b21-456d-b7d3-e43fe02a9ec8-bundle\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:44:54.115630 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:54.115592    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42" event={"ID":"2cae6749-1b21-456d-b7d3-e43fe02a9ec8","Type":"ContainerDied","Data":"cca7eea714b9c4c8dcd90f943e76020f560ec209ecff671452fe0207ad6646dd"}
Sep 18 18:44:54.115791 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:54.115639    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hx42"
Sep 18 18:44:54.115791 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:44:54.115682    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cca7eea714b9c4c8dcd90f943e76020f560ec209ecff671452fe0207ad6646dd"
Sep 18 18:45:02.033553 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.033518    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzzzd"]
Sep 18 18:45:02.033964 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.033824    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cae6749-1b21-456d-b7d3-e43fe02a9ec8" containerName="pull"
Sep 18 18:45:02.033964 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.033835    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cae6749-1b21-456d-b7d3-e43fe02a9ec8" containerName="pull"
Sep 18 18:45:02.033964 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.033850    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cae6749-1b21-456d-b7d3-e43fe02a9ec8" containerName="extract"
Sep 18 18:45:02.033964 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.033855    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cae6749-1b21-456d-b7d3-e43fe02a9ec8" containerName="extract"
Sep 18 18:45:02.033964 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.033865    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cae6749-1b21-456d-b7d3-e43fe02a9ec8" containerName="util"
Sep 18 18:45:02.033964 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.033870    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cae6749-1b21-456d-b7d3-e43fe02a9ec8" containerName="util"
Sep 18 18:45:02.033964 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.033917    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cae6749-1b21-456d-b7d3-e43fe02a9ec8" containerName="extract"
Sep 18 18:45:02.073098 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.073062    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzzzd"]
Sep 18 18:45:02.073267 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.073157    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzzzd"
Sep 18 18:45:02.075490 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.075466    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 18:45:02.075602 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.075561    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 18:45:02.075680 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.075632    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-mhc4b\""
Sep 18 18:45:02.106798 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.106765    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eba37969-cd6d-4aa6-9307-345a0d5133d9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wzzzd\" (UID: \"eba37969-cd6d-4aa6-9307-345a0d5133d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzzzd"
Sep 18 18:45:02.106939 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.106812    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69vzl\" (UniqueName: \"kubernetes.io/projected/eba37969-cd6d-4aa6-9307-345a0d5133d9-kube-api-access-69vzl\") pod \"openshift-lws-operator-fd8ccff4c-wzzzd\" (UID: \"eba37969-cd6d-4aa6-9307-345a0d5133d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzzzd"
Sep 18 18:45:02.208204 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.208159    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eba37969-cd6d-4aa6-9307-345a0d5133d9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wzzzd\" (UID: \"eba37969-cd6d-4aa6-9307-345a0d5133d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzzzd"
Sep 18 18:45:02.208354 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.208222    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69vzl\" (UniqueName: \"kubernetes.io/projected/eba37969-cd6d-4aa6-9307-345a0d5133d9-kube-api-access-69vzl\") pod \"openshift-lws-operator-fd8ccff4c-wzzzd\" (UID: \"eba37969-cd6d-4aa6-9307-345a0d5133d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzzzd"
Sep 18 18:45:02.208528 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.208509    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eba37969-cd6d-4aa6-9307-345a0d5133d9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wzzzd\" (UID: \"eba37969-cd6d-4aa6-9307-345a0d5133d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzzzd"
Sep 18 18:45:02.215693 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.215673    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69vzl\" (UniqueName: \"kubernetes.io/projected/eba37969-cd6d-4aa6-9307-345a0d5133d9-kube-api-access-69vzl\") pod \"openshift-lws-operator-fd8ccff4c-wzzzd\" (UID: \"eba37969-cd6d-4aa6-9307-345a0d5133d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzzzd"
Sep 18 18:45:02.382582 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.382550    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzzzd"
Sep 18 18:45:02.672490 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:02.672293    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzzzd"]
Sep 18 18:45:02.675446 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:45:02.675420    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeba37969_cd6d_4aa6_9307_345a0d5133d9.slice/crio-ddf713747f4a5787ab652c5ac2e1575b1aee43ed2ab94990fe38f3a10ab01286 WatchSource:0}: Error finding container ddf713747f4a5787ab652c5ac2e1575b1aee43ed2ab94990fe38f3a10ab01286: Status 404 returned error can't find the container with id ddf713747f4a5787ab652c5ac2e1575b1aee43ed2ab94990fe38f3a10ab01286
Sep 18 18:45:03.155994 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:03.155953    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzzzd" event={"ID":"eba37969-cd6d-4aa6-9307-345a0d5133d9","Type":"ContainerStarted","Data":"ddf713747f4a5787ab652c5ac2e1575b1aee43ed2ab94990fe38f3a10ab01286"}
Sep 18 18:45:05.167346 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:05.167306    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzzzd" event={"ID":"eba37969-cd6d-4aa6-9307-345a0d5133d9","Type":"ContainerStarted","Data":"948e1ab6982bfabd16fb8b7fffcf5e36f883ddb8e702f25325f214fba419069d"}
Sep 18 18:45:05.188725 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:05.188678    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzzzd" podStartSLOduration=1.073970238 podStartE2EDuration="3.188663016s" totalImagesPullingTime="2.114692778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:05.186679543 +0000 UTC m=+364.623258409" watchObservedRunningTime="2026-09-18 18:45:05.188663016 +0000 UTC m=+364.625241857"
Sep 18 18:45:09.798780 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:09.798747    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"]
Sep 18 18:45:09.802279 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:09.802263    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"
Sep 18 18:45:09.804414 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:09.804386    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ktfvw\""
Sep 18 18:45:09.804561 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:09.804473    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 18:45:09.804561 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:09.804481    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 18:45:09.810342 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:09.810320    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"]
Sep 18 18:45:09.870102 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:09.870072    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b341bf95-6f5c-4f21-b5ac-bbda1925c396-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5\" (UID: \"b341bf95-6f5c-4f21-b5ac-bbda1925c396\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"
Sep 18 18:45:09.870274 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:09.870113    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt8df\" (UniqueName: \"kubernetes.io/projected/b341bf95-6f5c-4f21-b5ac-bbda1925c396-kube-api-access-tt8df\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5\" (UID: \"b341bf95-6f5c-4f21-b5ac-bbda1925c396\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"
Sep 18 18:45:09.870274 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:09.870156    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b341bf95-6f5c-4f21-b5ac-bbda1925c396-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5\" (UID: \"b341bf95-6f5c-4f21-b5ac-bbda1925c396\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"
Sep 18 18:45:09.970769 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:09.970701    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b341bf95-6f5c-4f21-b5ac-bbda1925c396-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5\" (UID: \"b341bf95-6f5c-4f21-b5ac-bbda1925c396\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"
Sep 18 18:45:09.970943 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:09.970784    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt8df\" (UniqueName: \"kubernetes.io/projected/b341bf95-6f5c-4f21-b5ac-bbda1925c396-kube-api-access-tt8df\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5\" (UID: \"b341bf95-6f5c-4f21-b5ac-bbda1925c396\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"
Sep 18 18:45:09.970943 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:09.970852    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b341bf95-6f5c-4f21-b5ac-bbda1925c396-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5\" (UID: \"b341bf95-6f5c-4f21-b5ac-bbda1925c396\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"
Sep 18 18:45:09.971172 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:09.971150    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b341bf95-6f5c-4f21-b5ac-bbda1925c396-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5\" (UID: \"b341bf95-6f5c-4f21-b5ac-bbda1925c396\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"
Sep 18 18:45:09.971172 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:09.971165    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b341bf95-6f5c-4f21-b5ac-bbda1925c396-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5\" (UID: \"b341bf95-6f5c-4f21-b5ac-bbda1925c396\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"
Sep 18 18:45:09.978699 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:09.978677    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt8df\" (UniqueName: \"kubernetes.io/projected/b341bf95-6f5c-4f21-b5ac-bbda1925c396-kube-api-access-tt8df\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5\" (UID: \"b341bf95-6f5c-4f21-b5ac-bbda1925c396\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"
Sep 18 18:45:10.112964 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:10.112932    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"
Sep 18 18:45:10.384652 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:10.384627    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"]
Sep 18 18:45:10.386453 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:45:10.386427    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb341bf95_6f5c_4f21_b5ac_bbda1925c396.slice/crio-a0e6c2ea4ab51d8100d5260dc5cd2c0ba12988bf568fb742f2afc4662634b401 WatchSource:0}: Error finding container a0e6c2ea4ab51d8100d5260dc5cd2c0ba12988bf568fb742f2afc4662634b401: Status 404 returned error can't find the container with id a0e6c2ea4ab51d8100d5260dc5cd2c0ba12988bf568fb742f2afc4662634b401
Sep 18 18:45:11.197791 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:11.197758    2558 generic.go:356] "Generic (PLEG): container finished" podID="b341bf95-6f5c-4f21-b5ac-bbda1925c396" containerID="a108b673d1c542760f5e0550c3f002c599619f3ce744454fcf173115231fa971" exitCode=0
Sep 18 18:45:11.198221 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:11.197824    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5" event={"ID":"b341bf95-6f5c-4f21-b5ac-bbda1925c396","Type":"ContainerDied","Data":"a108b673d1c542760f5e0550c3f002c599619f3ce744454fcf173115231fa971"}
Sep 18 18:45:11.198221 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:11.197864    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5" event={"ID":"b341bf95-6f5c-4f21-b5ac-bbda1925c396","Type":"ContainerStarted","Data":"a0e6c2ea4ab51d8100d5260dc5cd2c0ba12988bf568fb742f2afc4662634b401"}
Sep 18 18:45:12.204053 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:12.204017    2558 generic.go:356] "Generic (PLEG): container finished" podID="b341bf95-6f5c-4f21-b5ac-bbda1925c396" containerID="c1916a72426feb9ae3ff49b5952b8bf9bc2345947a8d6ff7b0504b6d271a0f57" exitCode=0
Sep 18 18:45:12.204493 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:12.204102    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5" event={"ID":"b341bf95-6f5c-4f21-b5ac-bbda1925c396","Type":"ContainerDied","Data":"c1916a72426feb9ae3ff49b5952b8bf9bc2345947a8d6ff7b0504b6d271a0f57"}
Sep 18 18:45:13.210398 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:13.210364    2558 generic.go:356] "Generic (PLEG): container finished" podID="b341bf95-6f5c-4f21-b5ac-bbda1925c396" containerID="41b08c63c390da73013602869ed32a45ee59b62cba492f9b96636321d1d714d2" exitCode=0
Sep 18 18:45:13.210848 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:13.210427    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5" event={"ID":"b341bf95-6f5c-4f21-b5ac-bbda1925c396","Type":"ContainerDied","Data":"41b08c63c390da73013602869ed32a45ee59b62cba492f9b96636321d1d714d2"}
Sep 18 18:45:14.341652 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:14.341631    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"
Sep 18 18:45:14.405739 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:14.405711    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b341bf95-6f5c-4f21-b5ac-bbda1925c396-util\") pod \"b341bf95-6f5c-4f21-b5ac-bbda1925c396\" (UID: \"b341bf95-6f5c-4f21-b5ac-bbda1925c396\") "
Sep 18 18:45:14.405893 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:14.405770    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt8df\" (UniqueName: \"kubernetes.io/projected/b341bf95-6f5c-4f21-b5ac-bbda1925c396-kube-api-access-tt8df\") pod \"b341bf95-6f5c-4f21-b5ac-bbda1925c396\" (UID: \"b341bf95-6f5c-4f21-b5ac-bbda1925c396\") "
Sep 18 18:45:14.405893 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:14.405812    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b341bf95-6f5c-4f21-b5ac-bbda1925c396-bundle\") pod \"b341bf95-6f5c-4f21-b5ac-bbda1925c396\" (UID: \"b341bf95-6f5c-4f21-b5ac-bbda1925c396\") "
Sep 18 18:45:14.406542 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:14.406505    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b341bf95-6f5c-4f21-b5ac-bbda1925c396-bundle" (OuterVolumeSpecName: "bundle") pod "b341bf95-6f5c-4f21-b5ac-bbda1925c396" (UID: "b341bf95-6f5c-4f21-b5ac-bbda1925c396"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:45:14.407949 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:14.407919    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b341bf95-6f5c-4f21-b5ac-bbda1925c396-kube-api-access-tt8df" (OuterVolumeSpecName: "kube-api-access-tt8df") pod "b341bf95-6f5c-4f21-b5ac-bbda1925c396" (UID: "b341bf95-6f5c-4f21-b5ac-bbda1925c396"). InnerVolumeSpecName "kube-api-access-tt8df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:45:14.411653 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:14.411627    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b341bf95-6f5c-4f21-b5ac-bbda1925c396-util" (OuterVolumeSpecName: "util") pod "b341bf95-6f5c-4f21-b5ac-bbda1925c396" (UID: "b341bf95-6f5c-4f21-b5ac-bbda1925c396"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:45:14.506755 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:14.506670    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt8df\" (UniqueName: \"kubernetes.io/projected/b341bf95-6f5c-4f21-b5ac-bbda1925c396-kube-api-access-tt8df\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:45:14.506755 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:14.506695    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b341bf95-6f5c-4f21-b5ac-bbda1925c396-bundle\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:45:14.506755 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:14.506704    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b341bf95-6f5c-4f21-b5ac-bbda1925c396-util\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:45:15.221017 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:15.220988    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5" event={"ID":"b341bf95-6f5c-4f21-b5ac-bbda1925c396","Type":"ContainerDied","Data":"a0e6c2ea4ab51d8100d5260dc5cd2c0ba12988bf568fb742f2afc4662634b401"}
Sep 18 18:45:15.221169 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:15.221036    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0e6c2ea4ab51d8100d5260dc5cd2c0ba12988bf568fb742f2afc4662634b401"
Sep 18 18:45:15.221169 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:15.220995    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrw7x5"
Sep 18 18:45:20.832966 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.832932    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"]
Sep 18 18:45:20.833427 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.833235    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b341bf95-6f5c-4f21-b5ac-bbda1925c396" containerName="pull"
Sep 18 18:45:20.833427 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.833248    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b341bf95-6f5c-4f21-b5ac-bbda1925c396" containerName="pull"
Sep 18 18:45:20.833427 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.833270    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b341bf95-6f5c-4f21-b5ac-bbda1925c396" containerName="extract"
Sep 18 18:45:20.833427 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.833275    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b341bf95-6f5c-4f21-b5ac-bbda1925c396" containerName="extract"
Sep 18 18:45:20.833427 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.833284    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b341bf95-6f5c-4f21-b5ac-bbda1925c396" containerName="util"
Sep 18 18:45:20.833427 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.833290    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b341bf95-6f5c-4f21-b5ac-bbda1925c396" containerName="util"
Sep 18 18:45:20.833427 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.833339    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b341bf95-6f5c-4f21-b5ac-bbda1925c396" containerName="extract"
Sep 18 18:45:20.836493 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.836476    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"
Sep 18 18:45:20.839617 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.839590    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 18:45:20.839948 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.839930    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ktfvw\""
Sep 18 18:45:20.840495 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.840477    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 18:45:20.854901 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.854878    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"]
Sep 18 18:45:20.959383 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.959350    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3961eb11-704d-49aa-8b4e-b390b9531b83-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl\" (UID: \"3961eb11-704d-49aa-8b4e-b390b9531b83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"
Sep 18 18:45:20.959383 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.959387    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2qk\" (UniqueName: \"kubernetes.io/projected/3961eb11-704d-49aa-8b4e-b390b9531b83-kube-api-access-7t2qk\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl\" (UID: \"3961eb11-704d-49aa-8b4e-b390b9531b83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"
Sep 18 18:45:20.959590 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:20.959464    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3961eb11-704d-49aa-8b4e-b390b9531b83-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl\" (UID: \"3961eb11-704d-49aa-8b4e-b390b9531b83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"
Sep 18 18:45:21.060663 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:21.060618    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3961eb11-704d-49aa-8b4e-b390b9531b83-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl\" (UID: \"3961eb11-704d-49aa-8b4e-b390b9531b83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"
Sep 18 18:45:21.060838 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:21.060729    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3961eb11-704d-49aa-8b4e-b390b9531b83-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl\" (UID: \"3961eb11-704d-49aa-8b4e-b390b9531b83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"
Sep 18 18:45:21.060838 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:21.060766    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2qk\" (UniqueName: \"kubernetes.io/projected/3961eb11-704d-49aa-8b4e-b390b9531b83-kube-api-access-7t2qk\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl\" (UID: \"3961eb11-704d-49aa-8b4e-b390b9531b83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"
Sep 18 18:45:21.061074 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:21.061040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3961eb11-704d-49aa-8b4e-b390b9531b83-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl\" (UID: \"3961eb11-704d-49aa-8b4e-b390b9531b83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"
Sep 18 18:45:21.061210 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:21.061073    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3961eb11-704d-49aa-8b4e-b390b9531b83-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl\" (UID: \"3961eb11-704d-49aa-8b4e-b390b9531b83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"
Sep 18 18:45:21.069698 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:21.069676    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2qk\" (UniqueName: \"kubernetes.io/projected/3961eb11-704d-49aa-8b4e-b390b9531b83-kube-api-access-7t2qk\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl\" (UID: \"3961eb11-704d-49aa-8b4e-b390b9531b83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"
Sep 18 18:45:21.145654 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:21.145628    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"
Sep 18 18:45:21.432779 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:21.432753    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"]
Sep 18 18:45:21.435398 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:45:21.435372    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3961eb11_704d_49aa_8b4e_b390b9531b83.slice/crio-d88b51c8b336d5dac2efa6b99c45c946e8ccb3767a9ea860808cd63254e6ec9d WatchSource:0}: Error finding container d88b51c8b336d5dac2efa6b99c45c946e8ccb3767a9ea860808cd63254e6ec9d: Status 404 returned error can't find the container with id d88b51c8b336d5dac2efa6b99c45c946e8ccb3767a9ea860808cd63254e6ec9d
Sep 18 18:45:22.254693 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.254659    2558 generic.go:356] "Generic (PLEG): container finished" podID="3961eb11-704d-49aa-8b4e-b390b9531b83" containerID="14fd6fcbf05e7a2160e460d6231e73e61fde749ef58c1b490ee092bc60e2a06e" exitCode=0
Sep 18 18:45:22.255070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.254749    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl" event={"ID":"3961eb11-704d-49aa-8b4e-b390b9531b83","Type":"ContainerDied","Data":"14fd6fcbf05e7a2160e460d6231e73e61fde749ef58c1b490ee092bc60e2a06e"}
Sep 18 18:45:22.255070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.254843    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl" event={"ID":"3961eb11-704d-49aa-8b4e-b390b9531b83","Type":"ContainerStarted","Data":"d88b51c8b336d5dac2efa6b99c45c946e8ccb3767a9ea860808cd63254e6ec9d"}
Sep 18 18:45:22.832320 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.832289    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"]
Sep 18 18:45:22.835747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.835727    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:22.840029 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.840011    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 18:45:22.843925 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.843906    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fnn27\""
Sep 18 18:45:22.844355 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.844330    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 18:45:22.844443 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.844354    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 18:45:22.844670 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.844657    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 18:45:22.856060 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.856033    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"]
Sep 18 18:45:22.976998 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.976974    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/839f364a-1d5c-4f8c-88f5-5f54f2d22f55-manifests\") pod \"opendatahub-operator-controller-manager-55bcff78dd-28hzb\" (UID: \"839f364a-1d5c-4f8c-88f5-5f54f2d22f55\") " pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:22.977103 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.977010    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/839f364a-1d5c-4f8c-88f5-5f54f2d22f55-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55bcff78dd-28hzb\" (UID: \"839f364a-1d5c-4f8c-88f5-5f54f2d22f55\") " pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:22.977103 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.977036    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87srj\" (UniqueName: \"kubernetes.io/projected/839f364a-1d5c-4f8c-88f5-5f54f2d22f55-kube-api-access-87srj\") pod \"opendatahub-operator-controller-manager-55bcff78dd-28hzb\" (UID: \"839f364a-1d5c-4f8c-88f5-5f54f2d22f55\") " pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:22.977103 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.977080    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/839f364a-1d5c-4f8c-88f5-5f54f2d22f55-tmp\") pod \"opendatahub-operator-controller-manager-55bcff78dd-28hzb\" (UID: \"839f364a-1d5c-4f8c-88f5-5f54f2d22f55\") " pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:22.977306 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:22.977106    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/839f364a-1d5c-4f8c-88f5-5f54f2d22f55-webhook-cert\") pod \"opendatahub-operator-controller-manager-55bcff78dd-28hzb\" (UID: \"839f364a-1d5c-4f8c-88f5-5f54f2d22f55\") " pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:23.078308 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:23.078274    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/839f364a-1d5c-4f8c-88f5-5f54f2d22f55-webhook-cert\") pod \"opendatahub-operator-controller-manager-55bcff78dd-28hzb\" (UID: \"839f364a-1d5c-4f8c-88f5-5f54f2d22f55\") " pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:23.078469 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:23.078327    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/839f364a-1d5c-4f8c-88f5-5f54f2d22f55-manifests\") pod \"opendatahub-operator-controller-manager-55bcff78dd-28hzb\" (UID: \"839f364a-1d5c-4f8c-88f5-5f54f2d22f55\") " pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:23.078533 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:23.078489    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/839f364a-1d5c-4f8c-88f5-5f54f2d22f55-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55bcff78dd-28hzb\" (UID: \"839f364a-1d5c-4f8c-88f5-5f54f2d22f55\") " pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:23.078590 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:23.078535    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87srj\" (UniqueName: \"kubernetes.io/projected/839f364a-1d5c-4f8c-88f5-5f54f2d22f55-kube-api-access-87srj\") pod \"opendatahub-operator-controller-manager-55bcff78dd-28hzb\" (UID: \"839f364a-1d5c-4f8c-88f5-5f54f2d22f55\") " pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:23.078727 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:23.078705    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/839f364a-1d5c-4f8c-88f5-5f54f2d22f55-tmp\") pod \"opendatahub-operator-controller-manager-55bcff78dd-28hzb\" (UID: \"839f364a-1d5c-4f8c-88f5-5f54f2d22f55\") " pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:23.078866 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:23.078847    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/839f364a-1d5c-4f8c-88f5-5f54f2d22f55-manifests\") pod \"opendatahub-operator-controller-manager-55bcff78dd-28hzb\" (UID: \"839f364a-1d5c-4f8c-88f5-5f54f2d22f55\") " pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:23.080685 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:23.080665    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/839f364a-1d5c-4f8c-88f5-5f54f2d22f55-tmp\") pod \"opendatahub-operator-controller-manager-55bcff78dd-28hzb\" (UID: \"839f364a-1d5c-4f8c-88f5-5f54f2d22f55\") " pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:23.080787 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:23.080726    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/839f364a-1d5c-4f8c-88f5-5f54f2d22f55-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55bcff78dd-28hzb\" (UID: \"839f364a-1d5c-4f8c-88f5-5f54f2d22f55\") " pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:23.080975 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:23.080957    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/839f364a-1d5c-4f8c-88f5-5f54f2d22f55-webhook-cert\") pod \"opendatahub-operator-controller-manager-55bcff78dd-28hzb\" (UID: \"839f364a-1d5c-4f8c-88f5-5f54f2d22f55\") " pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:23.087272 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:23.087251    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87srj\" (UniqueName: \"kubernetes.io/projected/839f364a-1d5c-4f8c-88f5-5f54f2d22f55-kube-api-access-87srj\") pod \"opendatahub-operator-controller-manager-55bcff78dd-28hzb\" (UID: \"839f364a-1d5c-4f8c-88f5-5f54f2d22f55\") " pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:23.147592 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:23.147560    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:23.261879 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:23.261847    2558 generic.go:356] "Generic (PLEG): container finished" podID="3961eb11-704d-49aa-8b4e-b390b9531b83" containerID="dfb065654875d8ba473fcc2001a1cf3aeb4dfb5c78128e286f69bd4917662bbd" exitCode=0
Sep 18 18:45:23.262242 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:23.261932    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl" event={"ID":"3961eb11-704d-49aa-8b4e-b390b9531b83","Type":"ContainerDied","Data":"dfb065654875d8ba473fcc2001a1cf3aeb4dfb5c78128e286f69bd4917662bbd"}
Sep 18 18:45:23.439170 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:23.439142    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"]
Sep 18 18:45:23.485983 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:45:23.485953    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839f364a_1d5c_4f8c_88f5_5f54f2d22f55.slice/crio-8c7863f9dda8cbe34f4c8d431ece3b6d9c0010a56f440568436a50ab6f60392b WatchSource:0}: Error finding container 8c7863f9dda8cbe34f4c8d431ece3b6d9c0010a56f440568436a50ab6f60392b: Status 404 returned error can't find the container with id 8c7863f9dda8cbe34f4c8d431ece3b6d9c0010a56f440568436a50ab6f60392b
Sep 18 18:45:24.272349 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:24.272314    2558 generic.go:356] "Generic (PLEG): container finished" podID="3961eb11-704d-49aa-8b4e-b390b9531b83" containerID="a3fb95b19d5927652d32c11350e684c86cd177e25405c1116a927067fafcfcf5" exitCode=0
Sep 18 18:45:24.272815 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:24.272411    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl" event={"ID":"3961eb11-704d-49aa-8b4e-b390b9531b83","Type":"ContainerDied","Data":"a3fb95b19d5927652d32c11350e684c86cd177e25405c1116a927067fafcfcf5"}
Sep 18 18:45:24.273854 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:24.273824    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb" event={"ID":"839f364a-1d5c-4f8c-88f5-5f54f2d22f55","Type":"ContainerStarted","Data":"8c7863f9dda8cbe34f4c8d431ece3b6d9c0010a56f440568436a50ab6f60392b"}
Sep 18 18:45:26.544320 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:26.544296    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"
Sep 18 18:45:26.712456 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:26.712428    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3961eb11-704d-49aa-8b4e-b390b9531b83-util\") pod \"3961eb11-704d-49aa-8b4e-b390b9531b83\" (UID: \"3961eb11-704d-49aa-8b4e-b390b9531b83\") "
Sep 18 18:45:26.712602 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:26.712465    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3961eb11-704d-49aa-8b4e-b390b9531b83-bundle\") pod \"3961eb11-704d-49aa-8b4e-b390b9531b83\" (UID: \"3961eb11-704d-49aa-8b4e-b390b9531b83\") "
Sep 18 18:45:26.712602 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:26.712493    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t2qk\" (UniqueName: \"kubernetes.io/projected/3961eb11-704d-49aa-8b4e-b390b9531b83-kube-api-access-7t2qk\") pod \"3961eb11-704d-49aa-8b4e-b390b9531b83\" (UID: \"3961eb11-704d-49aa-8b4e-b390b9531b83\") "
Sep 18 18:45:26.713488 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:26.713457    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3961eb11-704d-49aa-8b4e-b390b9531b83-bundle" (OuterVolumeSpecName: "bundle") pod "3961eb11-704d-49aa-8b4e-b390b9531b83" (UID: "3961eb11-704d-49aa-8b4e-b390b9531b83"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:45:26.714514 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:26.714481    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3961eb11-704d-49aa-8b4e-b390b9531b83-kube-api-access-7t2qk" (OuterVolumeSpecName: "kube-api-access-7t2qk") pod "3961eb11-704d-49aa-8b4e-b390b9531b83" (UID: "3961eb11-704d-49aa-8b4e-b390b9531b83"). InnerVolumeSpecName "kube-api-access-7t2qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:45:26.717747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:26.717728    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3961eb11-704d-49aa-8b4e-b390b9531b83-util" (OuterVolumeSpecName: "util") pod "3961eb11-704d-49aa-8b4e-b390b9531b83" (UID: "3961eb11-704d-49aa-8b4e-b390b9531b83"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:45:26.814055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:26.813976    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3961eb11-704d-49aa-8b4e-b390b9531b83-util\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:45:26.814055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:26.814001    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3961eb11-704d-49aa-8b4e-b390b9531b83-bundle\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:45:26.814055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:26.814011    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t2qk\" (UniqueName: \"kubernetes.io/projected/3961eb11-704d-49aa-8b4e-b390b9531b83-kube-api-access-7t2qk\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:45:27.291799 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:27.291765    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl" event={"ID":"3961eb11-704d-49aa-8b4e-b390b9531b83","Type":"ContainerDied","Data":"d88b51c8b336d5dac2efa6b99c45c946e8ccb3767a9ea860808cd63254e6ec9d"}
Sep 18 18:45:27.291799 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:27.291785    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l5lwl"
Sep 18 18:45:27.292013 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:27.291814    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d88b51c8b336d5dac2efa6b99c45c946e8ccb3767a9ea860808cd63254e6ec9d"
Sep 18 18:45:27.293529 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:27.293502    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb" event={"ID":"839f364a-1d5c-4f8c-88f5-5f54f2d22f55","Type":"ContainerStarted","Data":"923489a0532d4252bd9fd74810c73a7c21de9b25288e6a707c3b48d2ce8ee007"}
Sep 18 18:45:28.299098 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:28.299062    2558 generic.go:356] "Generic (PLEG): container finished" podID="839f364a-1d5c-4f8c-88f5-5f54f2d22f55" containerID="923489a0532d4252bd9fd74810c73a7c21de9b25288e6a707c3b48d2ce8ee007" exitCode=0
Sep 18 18:45:28.299579 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:28.299113    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb" event={"ID":"839f364a-1d5c-4f8c-88f5-5f54f2d22f55","Type":"ContainerDied","Data":"923489a0532d4252bd9fd74810c73a7c21de9b25288e6a707c3b48d2ce8ee007"}
Sep 18 18:45:29.305677 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:29.305644    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb" event={"ID":"839f364a-1d5c-4f8c-88f5-5f54f2d22f55","Type":"ContainerStarted","Data":"7f6ecbd52b7391bd54d1e3c2dcf2a0c301e507baac0b710cca5aa0eda7273b67"}
Sep 18 18:45:29.306049 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:29.305731    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:29.329733 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:29.329661    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb" podStartSLOduration=2.9046449389999998 podStartE2EDuration="7.329646434s" totalImagesPullingTime="3.421003834s" totalInitContainerRuntime="1.003997661s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:45:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:29.327956265 +0000 UTC m=+388.764535106" watchObservedRunningTime="2026-09-18 18:45:29.329646434 +0000 UTC m=+388.766225276"
Sep 18 18:45:40.312478 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.312448    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55bcff78dd-28hzb"
Sep 18 18:45:40.610272 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.610243    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"]
Sep 18 18:45:40.610547 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.610536    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3961eb11-704d-49aa-8b4e-b390b9531b83" containerName="pull"
Sep 18 18:45:40.610595 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.610549    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3961eb11-704d-49aa-8b4e-b390b9531b83" containerName="pull"
Sep 18 18:45:40.610595 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.610572    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3961eb11-704d-49aa-8b4e-b390b9531b83" containerName="extract"
Sep 18 18:45:40.610595 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.610577    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3961eb11-704d-49aa-8b4e-b390b9531b83" containerName="extract"
Sep 18 18:45:40.610595 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.610585    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3961eb11-704d-49aa-8b4e-b390b9531b83" containerName="util"
Sep 18 18:45:40.610595 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.610592    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3961eb11-704d-49aa-8b4e-b390b9531b83" containerName="util"
Sep 18 18:45:40.610730 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.610644    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3961eb11-704d-49aa-8b4e-b390b9531b83" containerName="extract"
Sep 18 18:45:40.741997 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.741970    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"]
Sep 18 18:45:40.742151 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.742056    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:40.744780 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.744743    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 18:45:40.744780 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.744747    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 18:45:40.744780 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.744743    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 18:45:40.744985 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.744774    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vdzdx\""
Sep 18 18:45:40.934013 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.933929    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vzks\" (UniqueName: \"kubernetes.io/projected/b1d17e93-f66b-491c-b6d5-42b9473f1caf-kube-api-access-2vzks\") pod \"lws-controller-manager-7cddfd44f8-zdz8r\" (UID: \"b1d17e93-f66b-491c-b6d5-42b9473f1caf\") " pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:40.934013 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.933972    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1d17e93-f66b-491c-b6d5-42b9473f1caf-cert\") pod \"lws-controller-manager-7cddfd44f8-zdz8r\" (UID: \"b1d17e93-f66b-491c-b6d5-42b9473f1caf\") " pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:40.934013 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.933999    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1d17e93-f66b-491c-b6d5-42b9473f1caf-metrics-cert\") pod \"lws-controller-manager-7cddfd44f8-zdz8r\" (UID: \"b1d17e93-f66b-491c-b6d5-42b9473f1caf\") " pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:40.934264 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:40.934074    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b1d17e93-f66b-491c-b6d5-42b9473f1caf-manager-config\") pod \"lws-controller-manager-7cddfd44f8-zdz8r\" (UID: \"b1d17e93-f66b-491c-b6d5-42b9473f1caf\") " pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:41.034939 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:41.034906    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1d17e93-f66b-491c-b6d5-42b9473f1caf-metrics-cert\") pod \"lws-controller-manager-7cddfd44f8-zdz8r\" (UID: \"b1d17e93-f66b-491c-b6d5-42b9473f1caf\") " pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:41.035067 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:41.034946    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b1d17e93-f66b-491c-b6d5-42b9473f1caf-manager-config\") pod \"lws-controller-manager-7cddfd44f8-zdz8r\" (UID: \"b1d17e93-f66b-491c-b6d5-42b9473f1caf\") " pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:41.035067 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:41.035012    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vzks\" (UniqueName: \"kubernetes.io/projected/b1d17e93-f66b-491c-b6d5-42b9473f1caf-kube-api-access-2vzks\") pod \"lws-controller-manager-7cddfd44f8-zdz8r\" (UID: \"b1d17e93-f66b-491c-b6d5-42b9473f1caf\") " pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:41.035067 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:41.035055    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1d17e93-f66b-491c-b6d5-42b9473f1caf-cert\") pod \"lws-controller-manager-7cddfd44f8-zdz8r\" (UID: \"b1d17e93-f66b-491c-b6d5-42b9473f1caf\") " pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:41.035774 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:41.035755    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b1d17e93-f66b-491c-b6d5-42b9473f1caf-manager-config\") pod \"lws-controller-manager-7cddfd44f8-zdz8r\" (UID: \"b1d17e93-f66b-491c-b6d5-42b9473f1caf\") " pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:41.037534 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:41.037502    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1d17e93-f66b-491c-b6d5-42b9473f1caf-metrics-cert\") pod \"lws-controller-manager-7cddfd44f8-zdz8r\" (UID: \"b1d17e93-f66b-491c-b6d5-42b9473f1caf\") " pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:41.037534 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:41.037513    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1d17e93-f66b-491c-b6d5-42b9473f1caf-cert\") pod \"lws-controller-manager-7cddfd44f8-zdz8r\" (UID: \"b1d17e93-f66b-491c-b6d5-42b9473f1caf\") " pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:41.044264 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:41.044240    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vzks\" (UniqueName: \"kubernetes.io/projected/b1d17e93-f66b-491c-b6d5-42b9473f1caf-kube-api-access-2vzks\") pod \"lws-controller-manager-7cddfd44f8-zdz8r\" (UID: \"b1d17e93-f66b-491c-b6d5-42b9473f1caf\") " pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:41.050952 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:41.050931    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:41.325420 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:41.325378    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"]
Sep 18 18:45:41.328096 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:45:41.328067    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d17e93_f66b_491c_b6d5_42b9473f1caf.slice/crio-dd2d53b2f5446b84ae1c40222d59e06116149e72381007c266a1956f33155b6b WatchSource:0}: Error finding container dd2d53b2f5446b84ae1c40222d59e06116149e72381007c266a1956f33155b6b: Status 404 returned error can't find the container with id dd2d53b2f5446b84ae1c40222d59e06116149e72381007c266a1956f33155b6b
Sep 18 18:45:41.363210 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:41.363158    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r" event={"ID":"b1d17e93-f66b-491c-b6d5-42b9473f1caf","Type":"ContainerStarted","Data":"dd2d53b2f5446b84ae1c40222d59e06116149e72381007c266a1956f33155b6b"}
Sep 18 18:45:43.375703 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:43.375661    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r" event={"ID":"b1d17e93-f66b-491c-b6d5-42b9473f1caf","Type":"ContainerStarted","Data":"2f73d879a40d40c32d0e53221160ba1eceb89a703ca1062a629be4e66350aa86"}
Sep 18 18:45:43.376127 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:43.375799    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:43.394676 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:43.394628    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r" podStartSLOduration=1.978385184 podStartE2EDuration="3.394611259s" totalImagesPullingTime="1.416226075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:43.39352417 +0000 UTC m=+402.830103012" watchObservedRunningTime="2026-09-18 18:45:43.394611259 +0000 UTC m=+402.831190102"
Sep 18 18:45:49.350272 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.350236    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"]
Sep 18 18:45:49.353880 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.353862    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.356331 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.356306    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 18:45:49.356454 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.356430    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-lkfmp\""
Sep 18 18:45:49.367957 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.367935    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"]
Sep 18 18:45:49.400095 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.400062    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.400262 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.400110    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gswqh\" (UniqueName: \"kubernetes.io/projected/b7249632-c595-409b-bad4-360fbacabdaf-kube-api-access-gswqh\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.400262 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.400145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.400262 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.400177    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.400262 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.400245    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.400433 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.400269    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.400433 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.400298    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7249632-c595-409b-bad4-360fbacabdaf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.400433 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.400358    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7249632-c595-409b-bad4-360fbacabdaf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.400433 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.400396    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7249632-c595-409b-bad4-360fbacabdaf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.501247 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.501211    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7249632-c595-409b-bad4-360fbacabdaf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.501407 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.501265    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7249632-c595-409b-bad4-360fbacabdaf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.501407 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.501304    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.501407 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.501336    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gswqh\" (UniqueName: \"kubernetes.io/projected/b7249632-c595-409b-bad4-360fbacabdaf-kube-api-access-gswqh\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.501407 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.501361    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.501407 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.501389    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.501713 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.501431    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.501713 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.501456    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.501713 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.501481    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7249632-c595-409b-bad4-360fbacabdaf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.501875 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.501741    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.501875 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.501768    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.501875 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.501819    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.502078 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.502058    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.502209 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.502166    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7249632-c595-409b-bad4-360fbacabdaf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.503970 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.503949    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7249632-c595-409b-bad4-360fbacabdaf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.504099 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.504077    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.510634 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.510608    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7249632-c595-409b-bad4-360fbacabdaf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.510788 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.510764    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gswqh\" (UniqueName: \"kubernetes.io/projected/b7249632-c595-409b-bad4-360fbacabdaf-kube-api-access-gswqh\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bct8dwv\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.635855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.635771    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"]
Sep 18 18:45:49.640154 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.640137    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.658369 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.658345    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"]
Sep 18 18:45:49.664865 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.664846    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:49.703587 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.703552    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98840362-73e4-4d07-9b00-e2f02c2cb44f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.703777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.703597    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98840362-73e4-4d07-9b00-e2f02c2cb44f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.703777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.703623    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98840362-73e4-4d07-9b00-e2f02c2cb44f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.703777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.703642    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98840362-73e4-4d07-9b00-e2f02c2cb44f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.703777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.703663    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98840362-73e4-4d07-9b00-e2f02c2cb44f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.703777 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.703756    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98840362-73e4-4d07-9b00-e2f02c2cb44f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.704011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.703797    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98840362-73e4-4d07-9b00-e2f02c2cb44f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.704011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.703861    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98840362-73e4-4d07-9b00-e2f02c2cb44f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.704011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.703888    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpll4\" (UniqueName: \"kubernetes.io/projected/98840362-73e4-4d07-9b00-e2f02c2cb44f-kube-api-access-xpll4\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.804327 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.804278    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98840362-73e4-4d07-9b00-e2f02c2cb44f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.804327 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.804334    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpll4\" (UniqueName: \"kubernetes.io/projected/98840362-73e4-4d07-9b00-e2f02c2cb44f-kube-api-access-xpll4\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.804584 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.804372    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98840362-73e4-4d07-9b00-e2f02c2cb44f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.804584 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.804391    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98840362-73e4-4d07-9b00-e2f02c2cb44f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.804584 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.804414    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98840362-73e4-4d07-9b00-e2f02c2cb44f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.804584 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.804433    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98840362-73e4-4d07-9b00-e2f02c2cb44f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.804584 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.804448    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98840362-73e4-4d07-9b00-e2f02c2cb44f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.804584 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.804500    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98840362-73e4-4d07-9b00-e2f02c2cb44f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.804873 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.804673    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98840362-73e4-4d07-9b00-e2f02c2cb44f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.804873 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.804754    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98840362-73e4-4d07-9b00-e2f02c2cb44f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.804873 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.804791    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98840362-73e4-4d07-9b00-e2f02c2cb44f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.804873 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.804810    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98840362-73e4-4d07-9b00-e2f02c2cb44f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.805077 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.804965    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98840362-73e4-4d07-9b00-e2f02c2cb44f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.805492 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.805465    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98840362-73e4-4d07-9b00-e2f02c2cb44f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.806850 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.806829    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98840362-73e4-4d07-9b00-e2f02c2cb44f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.806948 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.806933    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98840362-73e4-4d07-9b00-e2f02c2cb44f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.811685 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.811664    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98840362-73e4-4d07-9b00-e2f02c2cb44f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.812154 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.812131    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpll4\" (UniqueName: \"kubernetes.io/projected/98840362-73e4-4d07-9b00-e2f02c2cb44f-kube-api-access-xpll4\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw\" (UID: \"98840362-73e4-4d07-9b00-e2f02c2cb44f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:49.940678 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.940609    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"]
Sep 18 18:45:49.942936 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:45:49.942908    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7249632_c595_409b_bad4_360fbacabdaf.slice/crio-59fd4f6c2af3eb1a17d0020e02fe2a09a5e0586531b1e117bab74160497921ba WatchSource:0}: Error finding container 59fd4f6c2af3eb1a17d0020e02fe2a09a5e0586531b1e117bab74160497921ba: Status 404 returned error can't find the container with id 59fd4f6c2af3eb1a17d0020e02fe2a09a5e0586531b1e117bab74160497921ba
Sep 18 18:45:49.950627 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:49.950597    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:50.223779 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:50.223751    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"]
Sep 18 18:45:50.225239 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:45:50.225213    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98840362_73e4_4d07_9b00_e2f02c2cb44f.slice/crio-f753023ff650260ccf887c84cfcb0e870d4130b7baadbc7ac38743abebcdf02c WatchSource:0}: Error finding container f753023ff650260ccf887c84cfcb0e870d4130b7baadbc7ac38743abebcdf02c: Status 404 returned error can't find the container with id f753023ff650260ccf887c84cfcb0e870d4130b7baadbc7ac38743abebcdf02c
Sep 18 18:45:50.410349 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:50.410313    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw" event={"ID":"98840362-73e4-4d07-9b00-e2f02c2cb44f","Type":"ContainerStarted","Data":"f753023ff650260ccf887c84cfcb0e870d4130b7baadbc7ac38743abebcdf02c"}
Sep 18 18:45:50.411456 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:50.411435    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv" event={"ID":"b7249632-c595-409b-bad4-360fbacabdaf","Type":"ContainerStarted","Data":"59fd4f6c2af3eb1a17d0020e02fe2a09a5e0586531b1e117bab74160497921ba"}
Sep 18 18:45:53.011726 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:53.011677    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:45:53.012096 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:53.011768    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:45:53.012096 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:53.011799    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:45:53.018614 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:53.018577    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:45:53.018710 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:53.018690    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:45:53.018768 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:53.018753    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:45:53.432176 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:53.432129    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw" event={"ID":"98840362-73e4-4d07-9b00-e2f02c2cb44f","Type":"ContainerStarted","Data":"02d4b3bad4c0b42b4445362eb55b6b2422f6d971b0a6d389d8209be3e765832b"}
Sep 18 18:45:53.433966 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:53.433939    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv" event={"ID":"b7249632-c595-409b-bad4-360fbacabdaf","Type":"ContainerStarted","Data":"3ed5fe73e10ce0615a497a8ae28f4c94ba3da8e842b3c14c80869cf7532c9898"}
Sep 18 18:45:53.453638 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:53.453594    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw" podStartSLOduration=1.6625277710000002 podStartE2EDuration="4.45357878s" totalImagesPullingTime="2.791051009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:53.450814816 +0000 UTC m=+412.887393658" watchObservedRunningTime="2026-09-18 18:45:53.45357878 +0000 UTC m=+412.890157622"
Sep 18 18:45:53.470766 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:53.470703    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv" podStartSLOduration=1.404084291 podStartE2EDuration="4.470685643s" totalImagesPullingTime="3.066601352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:53.468078646 +0000 UTC m=+412.904657488" watchObservedRunningTime="2026-09-18 18:45:53.470685643 +0000 UTC m=+412.907264486"
Sep 18 18:45:53.665504 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:53.665472    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:45:53.951471 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:53.951428    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:54.382934 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:54.382907    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7cddfd44f8-zdz8r"
Sep 18 18:45:54.667039 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:54.666947    2558 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 18 18:45:54.667039 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:54.667008    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv" podUID="b7249632-c595-409b-bad4-360fbacabdaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 18 18:45:54.955419 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:54.955343    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:55.445331 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:55.445296    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:55.446065 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:55.446046    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw"
Sep 18 18:45:55.495168 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:55.495132    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"]
Sep 18 18:45:55.666115 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:55.666076    2558 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 18 18:45:55.666305 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:55.666132    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv" podUID="b7249632-c595-409b-bad4-360fbacabdaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 18 18:45:56.451822 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:45:56.451775    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv" podUID="b7249632-c595-409b-bad4-360fbacabdaf" containerName="istio-proxy" containerID="cri-o://3ed5fe73e10ce0615a497a8ae28f4c94ba3da8e842b3c14c80869cf7532c9898" gracePeriod=30
Sep 18 18:46:01.712456 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.712398    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:46:01.815567 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.815531    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-workload-certs\") pod \"b7249632-c595-409b-bad4-360fbacabdaf\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") "
Sep 18 18:46:01.815727 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.815576    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-credential-socket\") pod \"b7249632-c595-409b-bad4-360fbacabdaf\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") "
Sep 18 18:46:01.815727 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.815602    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-istio-envoy\") pod \"b7249632-c595-409b-bad4-360fbacabdaf\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") "
Sep 18 18:46:01.815727 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.815630    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-workload-socket\") pod \"b7249632-c595-409b-bad4-360fbacabdaf\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") "
Sep 18 18:46:01.815727 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.815673    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-istio-data\") pod \"b7249632-c595-409b-bad4-360fbacabdaf\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") "
Sep 18 18:46:01.815727 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.815709    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gswqh\" (UniqueName: \"kubernetes.io/projected/b7249632-c595-409b-bad4-360fbacabdaf-kube-api-access-gswqh\") pod \"b7249632-c595-409b-bad4-360fbacabdaf\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") "
Sep 18 18:46:01.815977 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.815761    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7249632-c595-409b-bad4-360fbacabdaf-istio-podinfo\") pod \"b7249632-c595-409b-bad4-360fbacabdaf\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") "
Sep 18 18:46:01.815977 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.815810    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7249632-c595-409b-bad4-360fbacabdaf-istiod-ca-cert\") pod \"b7249632-c595-409b-bad4-360fbacabdaf\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") "
Sep 18 18:46:01.815977 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.815872    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7249632-c595-409b-bad4-360fbacabdaf-istio-token\") pod \"b7249632-c595-409b-bad4-360fbacabdaf\" (UID: \"b7249632-c595-409b-bad4-360fbacabdaf\") "
Sep 18 18:46:01.815977 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.815872    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b7249632-c595-409b-bad4-360fbacabdaf" (UID: "b7249632-c595-409b-bad4-360fbacabdaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:46:01.816165 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.815987    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b7249632-c595-409b-bad4-360fbacabdaf" (UID: "b7249632-c595-409b-bad4-360fbacabdaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:46:01.816165 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.816030    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-istio-data" (OuterVolumeSpecName: "istio-data") pod "b7249632-c595-409b-bad4-360fbacabdaf" (UID: "b7249632-c595-409b-bad4-360fbacabdaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:46:01.816165 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.816103    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b7249632-c595-409b-bad4-360fbacabdaf" (UID: "b7249632-c595-409b-bad4-360fbacabdaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:46:01.816332 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.816229    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:46:01.816332 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.816241    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7249632-c595-409b-bad4-360fbacabdaf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b7249632-c595-409b-bad4-360fbacabdaf" (UID: "b7249632-c595-409b-bad4-360fbacabdaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:46:01.816332 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.816248    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:46:01.816332 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.816284    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:46:01.816332 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.816299    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:46:01.818017 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.817984    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7249632-c595-409b-bad4-360fbacabdaf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b7249632-c595-409b-bad4-360fbacabdaf" (UID: "b7249632-c595-409b-bad4-360fbacabdaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:46:01.818140 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.818037    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b7249632-c595-409b-bad4-360fbacabdaf" (UID: "b7249632-c595-409b-bad4-360fbacabdaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:46:01.818140 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.818131    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7249632-c595-409b-bad4-360fbacabdaf-istio-token" (OuterVolumeSpecName: "istio-token") pod "b7249632-c595-409b-bad4-360fbacabdaf" (UID: "b7249632-c595-409b-bad4-360fbacabdaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:46:01.818257 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.818131    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7249632-c595-409b-bad4-360fbacabdaf-kube-api-access-gswqh" (OuterVolumeSpecName: "kube-api-access-gswqh") pod "b7249632-c595-409b-bad4-360fbacabdaf" (UID: "b7249632-c595-409b-bad4-360fbacabdaf"). InnerVolumeSpecName "kube-api-access-gswqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:46:01.916696 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.916663    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7249632-c595-409b-bad4-360fbacabdaf-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:46:01.916696 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.916688    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7249632-c595-409b-bad4-360fbacabdaf-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:46:01.916696 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.916697    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7249632-c595-409b-bad4-360fbacabdaf-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:46:01.916696 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.916705    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7249632-c595-409b-bad4-360fbacabdaf-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:46:01.916942 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:01.916714    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gswqh\" (UniqueName: \"kubernetes.io/projected/b7249632-c595-409b-bad4-360fbacabdaf-kube-api-access-gswqh\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:46:02.487263 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:02.487229    2558 generic.go:356] "Generic (PLEG): container finished" podID="b7249632-c595-409b-bad4-360fbacabdaf" containerID="3ed5fe73e10ce0615a497a8ae28f4c94ba3da8e842b3c14c80869cf7532c9898" exitCode=0
Sep 18 18:46:02.487426 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:02.487271    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"
Sep 18 18:46:02.487426 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:02.487315    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv" event={"ID":"b7249632-c595-409b-bad4-360fbacabdaf","Type":"ContainerDied","Data":"3ed5fe73e10ce0615a497a8ae28f4c94ba3da8e842b3c14c80869cf7532c9898"}
Sep 18 18:46:02.487426 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:02.487397    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv" event={"ID":"b7249632-c595-409b-bad4-360fbacabdaf","Type":"ContainerDied","Data":"59fd4f6c2af3eb1a17d0020e02fe2a09a5e0586531b1e117bab74160497921ba"}
Sep 18 18:46:02.487426 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:02.487421    2558 scope.go:117] "RemoveContainer" containerID="3ed5fe73e10ce0615a497a8ae28f4c94ba3da8e842b3c14c80869cf7532c9898"
Sep 18 18:46:02.497231 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:02.497211    2558 scope.go:117] "RemoveContainer" containerID="3ed5fe73e10ce0615a497a8ae28f4c94ba3da8e842b3c14c80869cf7532c9898"
Sep 18 18:46:02.497511 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:46:02.497492    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed5fe73e10ce0615a497a8ae28f4c94ba3da8e842b3c14c80869cf7532c9898\": container with ID starting with 3ed5fe73e10ce0615a497a8ae28f4c94ba3da8e842b3c14c80869cf7532c9898 not found: ID does not exist" containerID="3ed5fe73e10ce0615a497a8ae28f4c94ba3da8e842b3c14c80869cf7532c9898"
Sep 18 18:46:02.497562 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:02.497520    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed5fe73e10ce0615a497a8ae28f4c94ba3da8e842b3c14c80869cf7532c9898"} err="failed to get container status \"3ed5fe73e10ce0615a497a8ae28f4c94ba3da8e842b3c14c80869cf7532c9898\": rpc error: code = NotFound desc = could not find container \"3ed5fe73e10ce0615a497a8ae28f4c94ba3da8e842b3c14c80869cf7532c9898\": container with ID starting with 3ed5fe73e10ce0615a497a8ae28f4c94ba3da8e842b3c14c80869cf7532c9898 not found: ID does not exist"
Sep 18 18:46:02.521820 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:02.521792    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"]
Sep 18 18:46:02.525152 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:02.525132    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bct8dwv"]
Sep 18 18:46:02.574596 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:02.574567    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7249632-c595-409b-bad4-360fbacabdaf" path="/var/lib/kubelet/pods/b7249632-c595-409b-bad4-360fbacabdaf/volumes"
Sep 18 18:46:03.943728 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:03.943698    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw"]
Sep 18 18:46:03.944089 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:03.944019    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7249632-c595-409b-bad4-360fbacabdaf" containerName="istio-proxy"
Sep 18 18:46:03.944089 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:03.944032    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7249632-c595-409b-bad4-360fbacabdaf" containerName="istio-proxy"
Sep 18 18:46:03.944089 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:03.944089    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7249632-c595-409b-bad4-360fbacabdaf" containerName="istio-proxy"
Sep 18 18:46:03.954022 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:03.953995    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw"
Sep 18 18:46:03.961701 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:03.961664    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw"]
Sep 18 18:46:03.969858 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:03.969835    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-58pc9\""
Sep 18 18:46:03.971146 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:03.971124    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 18 18:46:04.036145 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:04.036111    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b75c261-ec15-4001-a6da-ebef6ce55c62-cert\") pod \"kserve-controller-manager-d6f4b65b8-9cwgw\" (UID: \"3b75c261-ec15-4001-a6da-ebef6ce55c62\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw"
Sep 18 18:46:04.036321 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:04.036179    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ckr8\" (UniqueName: \"kubernetes.io/projected/3b75c261-ec15-4001-a6da-ebef6ce55c62-kube-api-access-7ckr8\") pod \"kserve-controller-manager-d6f4b65b8-9cwgw\" (UID: \"3b75c261-ec15-4001-a6da-ebef6ce55c62\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw"
Sep 18 18:46:04.137289 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:04.137254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ckr8\" (UniqueName: \"kubernetes.io/projected/3b75c261-ec15-4001-a6da-ebef6ce55c62-kube-api-access-7ckr8\") pod \"kserve-controller-manager-d6f4b65b8-9cwgw\" (UID: \"3b75c261-ec15-4001-a6da-ebef6ce55c62\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw"
Sep 18 18:46:04.137482 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:04.137368    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b75c261-ec15-4001-a6da-ebef6ce55c62-cert\") pod \"kserve-controller-manager-d6f4b65b8-9cwgw\" (UID: \"3b75c261-ec15-4001-a6da-ebef6ce55c62\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw"
Sep 18 18:46:04.139925 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:04.139901    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b75c261-ec15-4001-a6da-ebef6ce55c62-cert\") pod \"kserve-controller-manager-d6f4b65b8-9cwgw\" (UID: \"3b75c261-ec15-4001-a6da-ebef6ce55c62\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw"
Sep 18 18:46:04.147461 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:04.147441    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ckr8\" (UniqueName: \"kubernetes.io/projected/3b75c261-ec15-4001-a6da-ebef6ce55c62-kube-api-access-7ckr8\") pod \"kserve-controller-manager-d6f4b65b8-9cwgw\" (UID: \"3b75c261-ec15-4001-a6da-ebef6ce55c62\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw"
Sep 18 18:46:04.267395 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:04.267309    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw"
Sep 18 18:46:04.569262 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:04.569232    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw"]
Sep 18 18:46:05.504311 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:05.504271    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw" event={"ID":"3b75c261-ec15-4001-a6da-ebef6ce55c62","Type":"ContainerStarted","Data":"b07e1c75de494512858d9c9f9b2e7648b578eb00be61e970b4108449aeb5403f"}
Sep 18 18:46:07.519132 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:07.519093    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw" event={"ID":"3b75c261-ec15-4001-a6da-ebef6ce55c62","Type":"ContainerStarted","Data":"5914c556361a612c88159d42fbe0ffeb7d35420bf5bf4fe8fcc761ea5cd9402b"}
Sep 18 18:46:07.519528 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:07.519167    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw"
Sep 18 18:46:07.540248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:07.540179    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw" podStartSLOduration=1.781056449 podStartE2EDuration="4.540163287s" totalImagesPullingTime="2.759106838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:46:07.538275446 +0000 UTC m=+426.974854289" watchObservedRunningTime="2026-09-18 18:46:07.540163287 +0000 UTC m=+426.976742128"
Sep 18 18:46:38.530390 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:38.530313    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9cwgw"
Sep 18 18:46:57.960266 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:57.960231    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"]
Sep 18 18:46:57.963804 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:57.963786    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"
Sep 18 18:46:57.966110 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:57.966093    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 18:46:57.966861 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:57.966840    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 18:46:57.966973 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:57.966844    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ktfvw\""
Sep 18 18:46:57.971054 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:57.971021    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"]
Sep 18 18:46:57.988517 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:57.988491    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6jdw\" (UniqueName: \"kubernetes.io/projected/ee8b25c5-4d85-41a9-a4de-728080cc831c-kube-api-access-s6jdw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww\" (UID: \"ee8b25c5-4d85-41a9-a4de-728080cc831c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"
Sep 18 18:46:57.988642 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:57.988542    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee8b25c5-4d85-41a9-a4de-728080cc831c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww\" (UID: \"ee8b25c5-4d85-41a9-a4de-728080cc831c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"
Sep 18 18:46:57.988642 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:57.988619    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee8b25c5-4d85-41a9-a4de-728080cc831c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww\" (UID: \"ee8b25c5-4d85-41a9-a4de-728080cc831c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"
Sep 18 18:46:58.063536 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.063499    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"]
Sep 18 18:46:58.067038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.067018    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"
Sep 18 18:46:58.075809 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.075787    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"]
Sep 18 18:46:58.089604 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.089576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee8b25c5-4d85-41a9-a4de-728080cc831c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww\" (UID: \"ee8b25c5-4d85-41a9-a4de-728080cc831c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"
Sep 18 18:46:58.089747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.089639    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee8b25c5-4d85-41a9-a4de-728080cc831c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww\" (UID: \"ee8b25c5-4d85-41a9-a4de-728080cc831c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"
Sep 18 18:46:58.089747 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.089723    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6jdw\" (UniqueName: \"kubernetes.io/projected/ee8b25c5-4d85-41a9-a4de-728080cc831c-kube-api-access-s6jdw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww\" (UID: \"ee8b25c5-4d85-41a9-a4de-728080cc831c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"
Sep 18 18:46:58.089976 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.089948    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee8b25c5-4d85-41a9-a4de-728080cc831c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww\" (UID: \"ee8b25c5-4d85-41a9-a4de-728080cc831c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"
Sep 18 18:46:58.090103 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.090083    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee8b25c5-4d85-41a9-a4de-728080cc831c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww\" (UID: \"ee8b25c5-4d85-41a9-a4de-728080cc831c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"
Sep 18 18:46:58.097833 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.097810    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6jdw\" (UniqueName: \"kubernetes.io/projected/ee8b25c5-4d85-41a9-a4de-728080cc831c-kube-api-access-s6jdw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww\" (UID: \"ee8b25c5-4d85-41a9-a4de-728080cc831c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"
Sep 18 18:46:58.190903 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.190859    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77446473-279a-4417-986c-d2b1d405d2e4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx\" (UID: \"77446473-279a-4417-986c-d2b1d405d2e4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"
Sep 18 18:46:58.191116 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.190944    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6xb\" (UniqueName: \"kubernetes.io/projected/77446473-279a-4417-986c-d2b1d405d2e4-kube-api-access-bt6xb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx\" (UID: \"77446473-279a-4417-986c-d2b1d405d2e4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"
Sep 18 18:46:58.191116 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.191057    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77446473-279a-4417-986c-d2b1d405d2e4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx\" (UID: \"77446473-279a-4417-986c-d2b1d405d2e4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"
Sep 18 18:46:58.193377 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.193351    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"]
Sep 18 18:46:58.197018 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.197000    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"
Sep 18 18:46:58.206800 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.206776    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"]
Sep 18 18:46:58.272331 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.272253    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"]
Sep 18 18:46:58.273327 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.273309    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"
Sep 18 18:46:58.276403 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.276388    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"
Sep 18 18:46:58.292793 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.292766    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77446473-279a-4417-986c-d2b1d405d2e4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx\" (UID: \"77446473-279a-4417-986c-d2b1d405d2e4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"
Sep 18 18:46:58.293418 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.293048    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6dzs\" (UniqueName: \"kubernetes.io/projected/3e5d9760-0ecc-4676-bb36-7bb0b721d268-kube-api-access-p6dzs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg\" (UID: \"3e5d9760-0ecc-4676-bb36-7bb0b721d268\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"
Sep 18 18:46:58.293615 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.293594    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6xb\" (UniqueName: \"kubernetes.io/projected/77446473-279a-4417-986c-d2b1d405d2e4-kube-api-access-bt6xb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx\" (UID: \"77446473-279a-4417-986c-d2b1d405d2e4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"
Sep 18 18:46:58.293742 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.293728    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e5d9760-0ecc-4676-bb36-7bb0b721d268-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg\" (UID: \"3e5d9760-0ecc-4676-bb36-7bb0b721d268\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"
Sep 18 18:46:58.293851 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.293838    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77446473-279a-4417-986c-d2b1d405d2e4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx\" (UID: \"77446473-279a-4417-986c-d2b1d405d2e4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"
Sep 18 18:46:58.294013 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.293999    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e5d9760-0ecc-4676-bb36-7bb0b721d268-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg\" (UID: \"3e5d9760-0ecc-4676-bb36-7bb0b721d268\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"
Sep 18 18:46:58.294147 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.293510    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77446473-279a-4417-986c-d2b1d405d2e4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx\" (UID: \"77446473-279a-4417-986c-d2b1d405d2e4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"
Sep 18 18:46:58.294289 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.294149    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77446473-279a-4417-986c-d2b1d405d2e4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx\" (UID: \"77446473-279a-4417-986c-d2b1d405d2e4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"
Sep 18 18:46:58.295596 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.295576    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"]
Sep 18 18:46:58.306364 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.306338    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6xb\" (UniqueName: \"kubernetes.io/projected/77446473-279a-4417-986c-d2b1d405d2e4-kube-api-access-bt6xb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx\" (UID: \"77446473-279a-4417-986c-d2b1d405d2e4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"
Sep 18 18:46:58.376019 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.375987    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"
Sep 18 18:46:58.395466 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.395419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6dzs\" (UniqueName: \"kubernetes.io/projected/3e5d9760-0ecc-4676-bb36-7bb0b721d268-kube-api-access-p6dzs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg\" (UID: \"3e5d9760-0ecc-4676-bb36-7bb0b721d268\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"
Sep 18 18:46:58.395583 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.395490    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e5d9760-0ecc-4676-bb36-7bb0b721d268-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg\" (UID: \"3e5d9760-0ecc-4676-bb36-7bb0b721d268\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"
Sep 18 18:46:58.395583 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.395532    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwvcj\" (UniqueName: \"kubernetes.io/projected/16d93403-de0a-45af-902f-8f3e3a7159bc-kube-api-access-qwvcj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft\" (UID: \"16d93403-de0a-45af-902f-8f3e3a7159bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"
Sep 18 18:46:58.395583 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.395571    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16d93403-de0a-45af-902f-8f3e3a7159bc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft\" (UID: \"16d93403-de0a-45af-902f-8f3e3a7159bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"
Sep 18 18:46:58.395723 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.395678    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16d93403-de0a-45af-902f-8f3e3a7159bc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft\" (UID: \"16d93403-de0a-45af-902f-8f3e3a7159bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"
Sep 18 18:46:58.395769 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.395734    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e5d9760-0ecc-4676-bb36-7bb0b721d268-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg\" (UID: \"3e5d9760-0ecc-4676-bb36-7bb0b721d268\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"
Sep 18 18:46:58.395971 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.395946    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e5d9760-0ecc-4676-bb36-7bb0b721d268-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg\" (UID: \"3e5d9760-0ecc-4676-bb36-7bb0b721d268\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"
Sep 18 18:46:58.396061 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.396041    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e5d9760-0ecc-4676-bb36-7bb0b721d268-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg\" (UID: \"3e5d9760-0ecc-4676-bb36-7bb0b721d268\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"
Sep 18 18:46:58.413558 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.413527    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6dzs\" (UniqueName: \"kubernetes.io/projected/3e5d9760-0ecc-4676-bb36-7bb0b721d268-kube-api-access-p6dzs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg\" (UID: \"3e5d9760-0ecc-4676-bb36-7bb0b721d268\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"
Sep 18 18:46:58.496258 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.496214    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwvcj\" (UniqueName: \"kubernetes.io/projected/16d93403-de0a-45af-902f-8f3e3a7159bc-kube-api-access-qwvcj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft\" (UID: \"16d93403-de0a-45af-902f-8f3e3a7159bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"
Sep 18 18:46:58.496258 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.496264    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16d93403-de0a-45af-902f-8f3e3a7159bc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft\" (UID: \"16d93403-de0a-45af-902f-8f3e3a7159bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"
Sep 18 18:46:58.496509 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.496297    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16d93403-de0a-45af-902f-8f3e3a7159bc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft\" (UID: \"16d93403-de0a-45af-902f-8f3e3a7159bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"
Sep 18 18:46:58.496665 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.496646    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16d93403-de0a-45af-902f-8f3e3a7159bc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft\" (UID: \"16d93403-de0a-45af-902f-8f3e3a7159bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"
Sep 18 18:46:58.496742 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.496713    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16d93403-de0a-45af-902f-8f3e3a7159bc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft\" (UID: \"16d93403-de0a-45af-902f-8f3e3a7159bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"
Sep 18 18:46:58.504897 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.504873    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwvcj\" (UniqueName: \"kubernetes.io/projected/16d93403-de0a-45af-902f-8f3e3a7159bc-kube-api-access-qwvcj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft\" (UID: \"16d93403-de0a-45af-902f-8f3e3a7159bc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"
Sep 18 18:46:58.507700 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.507683    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"
Sep 18 18:46:58.552994 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.552919    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"]
Sep 18 18:46:58.555982 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:46:58.555946    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8b25c5_4d85_41a9_a4de_728080cc831c.slice/crio-c21179450f25e4908c39cf46b656b0952a301e6eeee8d72d20315efdfbd858f8 WatchSource:0}: Error finding container c21179450f25e4908c39cf46b656b0952a301e6eeee8d72d20315efdfbd858f8: Status 404 returned error can't find the container with id c21179450f25e4908c39cf46b656b0952a301e6eeee8d72d20315efdfbd858f8
Sep 18 18:46:58.600100 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.600077    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"
Sep 18 18:46:58.655327 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.655240    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"]
Sep 18 18:46:58.662379 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:46:58.662348    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77446473_279a_4417_986c_d2b1d405d2e4.slice/crio-5d171421b3586294da84670505723c62987278268b2264d7370340c629968ccd WatchSource:0}: Error finding container 5d171421b3586294da84670505723c62987278268b2264d7370340c629968ccd: Status 404 returned error can't find the container with id 5d171421b3586294da84670505723c62987278268b2264d7370340c629968ccd
Sep 18 18:46:58.768112 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.768074    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx" event={"ID":"77446473-279a-4417-986c-d2b1d405d2e4","Type":"ContainerStarted","Data":"9ae26d2bfdcd75d3e83960e178a36acbd36460a1f7a015c5bc7a34d574e7858f"}
Sep 18 18:46:58.768347 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.768138    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx" event={"ID":"77446473-279a-4417-986c-d2b1d405d2e4","Type":"ContainerStarted","Data":"5d171421b3586294da84670505723c62987278268b2264d7370340c629968ccd"}
Sep 18 18:46:58.769868 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.769837    2558 generic.go:356] "Generic (PLEG): container finished" podID="ee8b25c5-4d85-41a9-a4de-728080cc831c" containerID="bff421e45183d33ea35e4332761e04e2283521ac262256646ed9507c3f1c7a62" exitCode=0
Sep 18 18:46:58.769969 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.769914    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww" event={"ID":"ee8b25c5-4d85-41a9-a4de-728080cc831c","Type":"ContainerDied","Data":"bff421e45183d33ea35e4332761e04e2283521ac262256646ed9507c3f1c7a62"}
Sep 18 18:46:58.769969 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.769963    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww" event={"ID":"ee8b25c5-4d85-41a9-a4de-728080cc831c","Type":"ContainerStarted","Data":"c21179450f25e4908c39cf46b656b0952a301e6eeee8d72d20315efdfbd858f8"}
Sep 18 18:46:58.785093 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.785066    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"]
Sep 18 18:46:58.816009 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:46:58.815943    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5d9760_0ecc_4676_bb36_7bb0b721d268.slice/crio-41dc53d9af8869804c9463b72fa808fa04b1f2c49b1adf53f5b56d938e81830c WatchSource:0}: Error finding container 41dc53d9af8869804c9463b72fa808fa04b1f2c49b1adf53f5b56d938e81830c: Status 404 returned error can't find the container with id 41dc53d9af8869804c9463b72fa808fa04b1f2c49b1adf53f5b56d938e81830c
Sep 18 18:46:58.881507 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:58.881470    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"]
Sep 18 18:46:58.888311 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:46:58.888271    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d93403_de0a_45af_902f_8f3e3a7159bc.slice/crio-1fb9fe27a280e1c8d8749e65e4fcf9cf006f2c903c28cd2e485d170803bd7985 WatchSource:0}: Error finding container 1fb9fe27a280e1c8d8749e65e4fcf9cf006f2c903c28cd2e485d170803bd7985: Status 404 returned error can't find the container with id 1fb9fe27a280e1c8d8749e65e4fcf9cf006f2c903c28cd2e485d170803bd7985
Sep 18 18:46:59.776834 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:59.776800    2558 generic.go:356] "Generic (PLEG): container finished" podID="3e5d9760-0ecc-4676-bb36-7bb0b721d268" containerID="182df8ce366359bc3292208a66bd995dfe85b80c0764ce6159ed095f703f99d4" exitCode=0
Sep 18 18:46:59.777299 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:59.776890    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg" event={"ID":"3e5d9760-0ecc-4676-bb36-7bb0b721d268","Type":"ContainerDied","Data":"182df8ce366359bc3292208a66bd995dfe85b80c0764ce6159ed095f703f99d4"}
Sep 18 18:46:59.777299 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:59.776952    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg" event={"ID":"3e5d9760-0ecc-4676-bb36-7bb0b721d268","Type":"ContainerStarted","Data":"41dc53d9af8869804c9463b72fa808fa04b1f2c49b1adf53f5b56d938e81830c"}
Sep 18 18:46:59.778888 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:59.778560    2558 generic.go:356] "Generic (PLEG): container finished" podID="77446473-279a-4417-986c-d2b1d405d2e4" containerID="9ae26d2bfdcd75d3e83960e178a36acbd36460a1f7a015c5bc7a34d574e7858f" exitCode=0
Sep 18 18:46:59.778888 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:59.778638    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx" event={"ID":"77446473-279a-4417-986c-d2b1d405d2e4","Type":"ContainerDied","Data":"9ae26d2bfdcd75d3e83960e178a36acbd36460a1f7a015c5bc7a34d574e7858f"}
Sep 18 18:46:59.780327 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:59.780303    2558 generic.go:356] "Generic (PLEG): container finished" podID="16d93403-de0a-45af-902f-8f3e3a7159bc" containerID="e19388fd2bb611985a81f256c0d750ab585d80ef2e6de642f8f0c9e0b89fb15f" exitCode=0
Sep 18 18:46:59.780425 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:59.780334    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft" event={"ID":"16d93403-de0a-45af-902f-8f3e3a7159bc","Type":"ContainerDied","Data":"e19388fd2bb611985a81f256c0d750ab585d80ef2e6de642f8f0c9e0b89fb15f"}
Sep 18 18:46:59.780425 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:46:59.780379    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft" event={"ID":"16d93403-de0a-45af-902f-8f3e3a7159bc","Type":"ContainerStarted","Data":"1fb9fe27a280e1c8d8749e65e4fcf9cf006f2c903c28cd2e485d170803bd7985"}
Sep 18 18:47:00.788361 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:00.788331    2558 generic.go:356] "Generic (PLEG): container finished" podID="3e5d9760-0ecc-4676-bb36-7bb0b721d268" containerID="4fce87ae625c7fa9e6e0a59ad82d85a815e93a0a23bf40d0cb5711497a3e0acf" exitCode=0
Sep 18 18:47:00.788748 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:00.788414    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg" event={"ID":"3e5d9760-0ecc-4676-bb36-7bb0b721d268","Type":"ContainerDied","Data":"4fce87ae625c7fa9e6e0a59ad82d85a815e93a0a23bf40d0cb5711497a3e0acf"}
Sep 18 18:47:00.790613 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:00.790521    2558 generic.go:356] "Generic (PLEG): container finished" podID="77446473-279a-4417-986c-d2b1d405d2e4" containerID="4006696a083137e5c20ccad2ac5159de46db0f5d793777aebb340eb3de3020fe" exitCode=0
Sep 18 18:47:00.790732 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:00.790594    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx" event={"ID":"77446473-279a-4417-986c-d2b1d405d2e4","Type":"ContainerDied","Data":"4006696a083137e5c20ccad2ac5159de46db0f5d793777aebb340eb3de3020fe"}
Sep 18 18:47:00.792605 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:00.792584    2558 generic.go:356] "Generic (PLEG): container finished" podID="16d93403-de0a-45af-902f-8f3e3a7159bc" containerID="848e0f3a71161fa15cb92bd07d6c833d5dea9153b9511e227cd6e582c53b3b40" exitCode=0
Sep 18 18:47:00.792681 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:00.792630    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft" event={"ID":"16d93403-de0a-45af-902f-8f3e3a7159bc","Type":"ContainerDied","Data":"848e0f3a71161fa15cb92bd07d6c833d5dea9153b9511e227cd6e582c53b3b40"}
Sep 18 18:47:01.800960 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:01.800868    2558 generic.go:356] "Generic (PLEG): container finished" podID="77446473-279a-4417-986c-d2b1d405d2e4" containerID="eccdab91c868a9740ebe99a229ab7185dc08f05b549ab8d86de2500d6524bf31" exitCode=0
Sep 18 18:47:01.800960 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:01.800945    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx" event={"ID":"77446473-279a-4417-986c-d2b1d405d2e4","Type":"ContainerDied","Data":"eccdab91c868a9740ebe99a229ab7185dc08f05b549ab8d86de2500d6524bf31"}
Sep 18 18:47:01.803181 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:01.803160    2558 generic.go:356] "Generic (PLEG): container finished" podID="16d93403-de0a-45af-902f-8f3e3a7159bc" containerID="c726acb341fd489be18b2702465b596786872d4472317de1138969595df62687" exitCode=0
Sep 18 18:47:01.803304 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:01.803220    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft" event={"ID":"16d93403-de0a-45af-902f-8f3e3a7159bc","Type":"ContainerDied","Data":"c726acb341fd489be18b2702465b596786872d4472317de1138969595df62687"}
Sep 18 18:47:01.805584 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:01.805565    2558 generic.go:356] "Generic (PLEG): container finished" podID="3e5d9760-0ecc-4676-bb36-7bb0b721d268" containerID="1240d6a75e9bde824fcabfcabdf3eab46e8e26d065527eb23b4c2efd2c618c10" exitCode=0
Sep 18 18:47:01.805672 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:01.805594    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg" event={"ID":"3e5d9760-0ecc-4676-bb36-7bb0b721d268","Type":"ContainerDied","Data":"1240d6a75e9bde824fcabfcabdf3eab46e8e26d065527eb23b4c2efd2c618c10"}
Sep 18 18:47:02.960949 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:02.960920    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"
Sep 18 18:47:03.013295 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.013272    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"
Sep 18 18:47:03.031733 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.031708    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e5d9760-0ecc-4676-bb36-7bb0b721d268-bundle\") pod \"3e5d9760-0ecc-4676-bb36-7bb0b721d268\" (UID: \"3e5d9760-0ecc-4676-bb36-7bb0b721d268\") "
Sep 18 18:47:03.031841 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.031755    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6dzs\" (UniqueName: \"kubernetes.io/projected/3e5d9760-0ecc-4676-bb36-7bb0b721d268-kube-api-access-p6dzs\") pod \"3e5d9760-0ecc-4676-bb36-7bb0b721d268\" (UID: \"3e5d9760-0ecc-4676-bb36-7bb0b721d268\") "
Sep 18 18:47:03.031841 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.031790    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e5d9760-0ecc-4676-bb36-7bb0b721d268-util\") pod \"3e5d9760-0ecc-4676-bb36-7bb0b721d268\" (UID: \"3e5d9760-0ecc-4676-bb36-7bb0b721d268\") "
Sep 18 18:47:03.032449 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.032419    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5d9760-0ecc-4676-bb36-7bb0b721d268-bundle" (OuterVolumeSpecName: "bundle") pod "3e5d9760-0ecc-4676-bb36-7bb0b721d268" (UID: "3e5d9760-0ecc-4676-bb36-7bb0b721d268"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:47:03.033988 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.033962    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5d9760-0ecc-4676-bb36-7bb0b721d268-kube-api-access-p6dzs" (OuterVolumeSpecName: "kube-api-access-p6dzs") pod "3e5d9760-0ecc-4676-bb36-7bb0b721d268" (UID: "3e5d9760-0ecc-4676-bb36-7bb0b721d268"). InnerVolumeSpecName "kube-api-access-p6dzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:47:03.037822 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.037797    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5d9760-0ecc-4676-bb36-7bb0b721d268-util" (OuterVolumeSpecName: "util") pod "3e5d9760-0ecc-4676-bb36-7bb0b721d268" (UID: "3e5d9760-0ecc-4676-bb36-7bb0b721d268"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:47:03.040783 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.040763    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"
Sep 18 18:47:03.132835 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.132806    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt6xb\" (UniqueName: \"kubernetes.io/projected/77446473-279a-4417-986c-d2b1d405d2e4-kube-api-access-bt6xb\") pod \"77446473-279a-4417-986c-d2b1d405d2e4\" (UID: \"77446473-279a-4417-986c-d2b1d405d2e4\") "
Sep 18 18:47:03.132940 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.132885    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77446473-279a-4417-986c-d2b1d405d2e4-bundle\") pod \"77446473-279a-4417-986c-d2b1d405d2e4\" (UID: \"77446473-279a-4417-986c-d2b1d405d2e4\") "
Sep 18 18:47:03.132940 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.132903    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16d93403-de0a-45af-902f-8f3e3a7159bc-util\") pod \"16d93403-de0a-45af-902f-8f3e3a7159bc\" (UID: \"16d93403-de0a-45af-902f-8f3e3a7159bc\") "
Sep 18 18:47:03.133055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.132941    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77446473-279a-4417-986c-d2b1d405d2e4-util\") pod \"77446473-279a-4417-986c-d2b1d405d2e4\" (UID: \"77446473-279a-4417-986c-d2b1d405d2e4\") "
Sep 18 18:47:03.133055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.132985    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16d93403-de0a-45af-902f-8f3e3a7159bc-bundle\") pod \"16d93403-de0a-45af-902f-8f3e3a7159bc\" (UID: \"16d93403-de0a-45af-902f-8f3e3a7159bc\") "
Sep 18 18:47:03.133055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.133009    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwvcj\" (UniqueName: \"kubernetes.io/projected/16d93403-de0a-45af-902f-8f3e3a7159bc-kube-api-access-qwvcj\") pod \"16d93403-de0a-45af-902f-8f3e3a7159bc\" (UID: \"16d93403-de0a-45af-902f-8f3e3a7159bc\") "
Sep 18 18:47:03.133284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.133226    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e5d9760-0ecc-4676-bb36-7bb0b721d268-bundle\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:03.133284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.133248    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6dzs\" (UniqueName: \"kubernetes.io/projected/3e5d9760-0ecc-4676-bb36-7bb0b721d268-kube-api-access-p6dzs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:03.133284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.133263    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e5d9760-0ecc-4676-bb36-7bb0b721d268-util\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:03.133511 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.133467    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77446473-279a-4417-986c-d2b1d405d2e4-bundle" (OuterVolumeSpecName: "bundle") pod "77446473-279a-4417-986c-d2b1d405d2e4" (UID: "77446473-279a-4417-986c-d2b1d405d2e4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:47:03.133831 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.133801    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d93403-de0a-45af-902f-8f3e3a7159bc-bundle" (OuterVolumeSpecName: "bundle") pod "16d93403-de0a-45af-902f-8f3e3a7159bc" (UID: "16d93403-de0a-45af-902f-8f3e3a7159bc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:47:03.135110 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.135080    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d93403-de0a-45af-902f-8f3e3a7159bc-kube-api-access-qwvcj" (OuterVolumeSpecName: "kube-api-access-qwvcj") pod "16d93403-de0a-45af-902f-8f3e3a7159bc" (UID: "16d93403-de0a-45af-902f-8f3e3a7159bc"). InnerVolumeSpecName "kube-api-access-qwvcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:47:03.135221 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.135158    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77446473-279a-4417-986c-d2b1d405d2e4-kube-api-access-bt6xb" (OuterVolumeSpecName: "kube-api-access-bt6xb") pod "77446473-279a-4417-986c-d2b1d405d2e4" (UID: "77446473-279a-4417-986c-d2b1d405d2e4"). InnerVolumeSpecName "kube-api-access-bt6xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:47:03.138842 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.138809    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77446473-279a-4417-986c-d2b1d405d2e4-util" (OuterVolumeSpecName: "util") pod "77446473-279a-4417-986c-d2b1d405d2e4" (UID: "77446473-279a-4417-986c-d2b1d405d2e4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:47:03.139149 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.139131    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d93403-de0a-45af-902f-8f3e3a7159bc-util" (OuterVolumeSpecName: "util") pod "16d93403-de0a-45af-902f-8f3e3a7159bc" (UID: "16d93403-de0a-45af-902f-8f3e3a7159bc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:47:03.234490 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.234461    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16d93403-de0a-45af-902f-8f3e3a7159bc-bundle\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:03.234490 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.234485    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwvcj\" (UniqueName: \"kubernetes.io/projected/16d93403-de0a-45af-902f-8f3e3a7159bc-kube-api-access-qwvcj\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:03.234490 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.234494    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt6xb\" (UniqueName: \"kubernetes.io/projected/77446473-279a-4417-986c-d2b1d405d2e4-kube-api-access-bt6xb\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:03.234686 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.234502    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77446473-279a-4417-986c-d2b1d405d2e4-bundle\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:03.234686 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.234510    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16d93403-de0a-45af-902f-8f3e3a7159bc-util\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:03.234686 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.234517    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77446473-279a-4417-986c-d2b1d405d2e4-util\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:03.817161 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.817132    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg"
Sep 18 18:47:03.817373 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.817131    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chvbbg" event={"ID":"3e5d9760-0ecc-4676-bb36-7bb0b721d268","Type":"ContainerDied","Data":"41dc53d9af8869804c9463b72fa808fa04b1f2c49b1adf53f5b56d938e81830c"}
Sep 18 18:47:03.817373 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.817240    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41dc53d9af8869804c9463b72fa808fa04b1f2c49b1adf53f5b56d938e81830c"
Sep 18 18:47:03.819127 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.819108    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx"
Sep 18 18:47:03.819127 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.819115    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77cqxx" event={"ID":"77446473-279a-4417-986c-d2b1d405d2e4","Type":"ContainerDied","Data":"5d171421b3586294da84670505723c62987278268b2264d7370340c629968ccd"}
Sep 18 18:47:03.819340 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.819150    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d171421b3586294da84670505723c62987278268b2264d7370340c629968ccd"
Sep 18 18:47:03.821150 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.821122    2558 generic.go:356] "Generic (PLEG): container finished" podID="ee8b25c5-4d85-41a9-a4de-728080cc831c" containerID="ad9f887376489889c590f8c34386cd4b26817e43d4651d356e1d78ca7755a8be" exitCode=0
Sep 18 18:47:03.821256 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.821214    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww" event={"ID":"ee8b25c5-4d85-41a9-a4de-728080cc831c","Type":"ContainerDied","Data":"ad9f887376489889c590f8c34386cd4b26817e43d4651d356e1d78ca7755a8be"}
Sep 18 18:47:03.823367 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.823340    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft" event={"ID":"16d93403-de0a-45af-902f-8f3e3a7159bc","Type":"ContainerDied","Data":"1fb9fe27a280e1c8d8749e65e4fcf9cf006f2c903c28cd2e485d170803bd7985"}
Sep 18 18:47:03.823465 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.823410    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fb9fe27a280e1c8d8749e65e4fcf9cf006f2c903c28cd2e485d170803bd7985"
Sep 18 18:47:03.823465 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:03.823449    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aflvft"
Sep 18 18:47:04.831492 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:04.831456    2558 generic.go:356] "Generic (PLEG): container finished" podID="ee8b25c5-4d85-41a9-a4de-728080cc831c" containerID="c585ff273fe0c3989fc7c8acf14e3e7d00e11e04bf0deada74b241a164c449bf" exitCode=0
Sep 18 18:47:04.831492 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:04.831497    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww" event={"ID":"ee8b25c5-4d85-41a9-a4de-728080cc831c","Type":"ContainerDied","Data":"c585ff273fe0c3989fc7c8acf14e3e7d00e11e04bf0deada74b241a164c449bf"}
Sep 18 18:47:05.972152 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:05.972131    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"
Sep 18 18:47:06.058094 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:06.058060    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee8b25c5-4d85-41a9-a4de-728080cc831c-bundle\") pod \"ee8b25c5-4d85-41a9-a4de-728080cc831c\" (UID: \"ee8b25c5-4d85-41a9-a4de-728080cc831c\") "
Sep 18 18:47:06.058279 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:06.058150    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee8b25c5-4d85-41a9-a4de-728080cc831c-util\") pod \"ee8b25c5-4d85-41a9-a4de-728080cc831c\" (UID: \"ee8b25c5-4d85-41a9-a4de-728080cc831c\") "
Sep 18 18:47:06.058279 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:06.058206    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6jdw\" (UniqueName: \"kubernetes.io/projected/ee8b25c5-4d85-41a9-a4de-728080cc831c-kube-api-access-s6jdw\") pod \"ee8b25c5-4d85-41a9-a4de-728080cc831c\" (UID: \"ee8b25c5-4d85-41a9-a4de-728080cc831c\") "
Sep 18 18:47:06.058635 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:06.058609    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8b25c5-4d85-41a9-a4de-728080cc831c-bundle" (OuterVolumeSpecName: "bundle") pod "ee8b25c5-4d85-41a9-a4de-728080cc831c" (UID: "ee8b25c5-4d85-41a9-a4de-728080cc831c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:47:06.060308 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:06.060287    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8b25c5-4d85-41a9-a4de-728080cc831c-kube-api-access-s6jdw" (OuterVolumeSpecName: "kube-api-access-s6jdw") pod "ee8b25c5-4d85-41a9-a4de-728080cc831c" (UID: "ee8b25c5-4d85-41a9-a4de-728080cc831c"). InnerVolumeSpecName "kube-api-access-s6jdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:47:06.064870 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:06.064821    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8b25c5-4d85-41a9-a4de-728080cc831c-util" (OuterVolumeSpecName: "util") pod "ee8b25c5-4d85-41a9-a4de-728080cc831c" (UID: "ee8b25c5-4d85-41a9-a4de-728080cc831c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:47:06.159216 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:06.159116    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee8b25c5-4d85-41a9-a4de-728080cc831c-util\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:06.159216 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:06.159142    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6jdw\" (UniqueName: \"kubernetes.io/projected/ee8b25c5-4d85-41a9-a4de-728080cc831c-kube-api-access-s6jdw\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:06.159216 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:06.159153    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee8b25c5-4d85-41a9-a4de-728080cc831c-bundle\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:06.843784 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:06.843753    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww" event={"ID":"ee8b25c5-4d85-41a9-a4de-728080cc831c","Type":"ContainerDied","Data":"c21179450f25e4908c39cf46b656b0952a301e6eeee8d72d20315efdfbd858f8"}
Sep 18 18:47:06.843943 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:06.843793    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qncww"
Sep 18 18:47:06.843943 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:06.843803    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c21179450f25e4908c39cf46b656b0952a301e6eeee8d72d20315efdfbd858f8"
Sep 18 18:47:11.773484 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.773441    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"]
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.773918    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77446473-279a-4417-986c-d2b1d405d2e4" containerName="util"
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.773937    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="77446473-279a-4417-986c-d2b1d405d2e4" containerName="util"
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.773958    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e5d9760-0ecc-4676-bb36-7bb0b721d268" containerName="extract"
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.773968    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e5d9760-0ecc-4676-bb36-7bb0b721d268" containerName="extract"
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.773980    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d93403-de0a-45af-902f-8f3e3a7159bc" containerName="util"
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.773990    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d93403-de0a-45af-902f-8f3e3a7159bc" containerName="util"
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774000    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e5d9760-0ecc-4676-bb36-7bb0b721d268" containerName="pull"
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774007    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e5d9760-0ecc-4676-bb36-7bb0b721d268" containerName="pull"
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774022    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77446473-279a-4417-986c-d2b1d405d2e4" containerName="pull"
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774030    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="77446473-279a-4417-986c-d2b1d405d2e4" containerName="pull"
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774040    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee8b25c5-4d85-41a9-a4de-728080cc831c" containerName="pull"
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774049    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee8b25c5-4d85-41a9-a4de-728080cc831c" containerName="pull"
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774061    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee8b25c5-4d85-41a9-a4de-728080cc831c" containerName="util"
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774069    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee8b25c5-4d85-41a9-a4de-728080cc831c" containerName="util"
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774077    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77446473-279a-4417-986c-d2b1d405d2e4" containerName="extract"
Sep 18 18:47:11.774080 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774085    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="77446473-279a-4417-986c-d2b1d405d2e4" containerName="extract"
Sep 18 18:47:11.774855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774095    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d93403-de0a-45af-902f-8f3e3a7159bc" containerName="pull"
Sep 18 18:47:11.774855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774102    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d93403-de0a-45af-902f-8f3e3a7159bc" containerName="pull"
Sep 18 18:47:11.774855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774116    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e5d9760-0ecc-4676-bb36-7bb0b721d268" containerName="util"
Sep 18 18:47:11.774855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774124    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e5d9760-0ecc-4676-bb36-7bb0b721d268" containerName="util"
Sep 18 18:47:11.774855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774136    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d93403-de0a-45af-902f-8f3e3a7159bc" containerName="extract"
Sep 18 18:47:11.774855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774144    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d93403-de0a-45af-902f-8f3e3a7159bc" containerName="extract"
Sep 18 18:47:11.774855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774156    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee8b25c5-4d85-41a9-a4de-728080cc831c" containerName="extract"
Sep 18 18:47:11.774855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774164    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee8b25c5-4d85-41a9-a4de-728080cc831c" containerName="extract"
Sep 18 18:47:11.774855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774308    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee8b25c5-4d85-41a9-a4de-728080cc831c" containerName="extract"
Sep 18 18:47:11.774855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774324    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e5d9760-0ecc-4676-bb36-7bb0b721d268" containerName="extract"
Sep 18 18:47:11.774855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774334    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="77446473-279a-4417-986c-d2b1d405d2e4" containerName="extract"
Sep 18 18:47:11.774855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.774344    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="16d93403-de0a-45af-902f-8f3e3a7159bc" containerName="extract"
Sep 18 18:47:11.777987 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.777966    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:11.780994 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.780967    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 18:47:11.781088 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.780989    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-ld8hw\""
Sep 18 18:47:11.781655 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.781639    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 18:47:11.789384 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.789362    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"]
Sep 18 18:47:11.909568 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:11.909531    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crd6c\" (UniqueName: \"kubernetes.io/projected/d5d8a066-20d8-4861-b7a6-dff4f76210fa-kube-api-access-crd6c\") pod \"limitador-operator-controller-manager-74495c69f8-xlzr4\" (UID: \"d5d8a066-20d8-4861-b7a6-dff4f76210fa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:12.010841 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:12.010801    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crd6c\" (UniqueName: \"kubernetes.io/projected/d5d8a066-20d8-4861-b7a6-dff4f76210fa-kube-api-access-crd6c\") pod \"limitador-operator-controller-manager-74495c69f8-xlzr4\" (UID: \"d5d8a066-20d8-4861-b7a6-dff4f76210fa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:12.019237 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:12.019214    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crd6c\" (UniqueName: \"kubernetes.io/projected/d5d8a066-20d8-4861-b7a6-dff4f76210fa-kube-api-access-crd6c\") pod \"limitador-operator-controller-manager-74495c69f8-xlzr4\" (UID: \"d5d8a066-20d8-4861-b7a6-dff4f76210fa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:12.089118 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:12.089090    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:12.363539 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:12.363514    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"]
Sep 18 18:47:12.366113 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:47:12.366081    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d8a066_20d8_4861_b7a6_dff4f76210fa.slice/crio-2b95614ab5d7ea242077dd2a2d1814c0b97708a8f02dcfdb0a35149dc842b0e0 WatchSource:0}: Error finding container 2b95614ab5d7ea242077dd2a2d1814c0b97708a8f02dcfdb0a35149dc842b0e0: Status 404 returned error can't find the container with id 2b95614ab5d7ea242077dd2a2d1814c0b97708a8f02dcfdb0a35149dc842b0e0
Sep 18 18:47:12.875354 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:12.875322    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4" event={"ID":"d5d8a066-20d8-4861-b7a6-dff4f76210fa","Type":"ContainerStarted","Data":"2b95614ab5d7ea242077dd2a2d1814c0b97708a8f02dcfdb0a35149dc842b0e0"}
Sep 18 18:47:14.889143 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:14.889044    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4" event={"ID":"d5d8a066-20d8-4861-b7a6-dff4f76210fa","Type":"ContainerStarted","Data":"ea686843382c30cbb5daa981c8fe86439e7891e2ad8466d8a8138a647414c0b1"}
Sep 18 18:47:14.889609 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:14.889203    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:14.913283 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:14.913230    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4" podStartSLOduration=1.671775089 podStartE2EDuration="3.913212541s" totalImagesPullingTime="2.241437452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:47:14.911156695 +0000 UTC m=+494.347735537" watchObservedRunningTime="2026-09-18 18:47:14.913212541 +0000 UTC m=+494.349791380"
Sep 18 18:47:16.424449 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:16.424415    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-27v92"]
Sep 18 18:47:16.428179 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:16.428153    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-27v92"
Sep 18 18:47:16.431071 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:16.431044    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-pmflp\""
Sep 18 18:47:16.431758 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:16.431742    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 18:47:16.451680 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:16.451659    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-27v92"]
Sep 18 18:47:16.548565 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:16.548530    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m6dd\" (UniqueName: \"kubernetes.io/projected/cf4cff2f-a2c3-4e90-b08c-3100811d01be-kube-api-access-9m6dd\") pod \"dns-operator-controller-manager-7b84cd7c9c-27v92\" (UID: \"cf4cff2f-a2c3-4e90-b08c-3100811d01be\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-27v92"
Sep 18 18:47:16.649097 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:16.649063    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m6dd\" (UniqueName: \"kubernetes.io/projected/cf4cff2f-a2c3-4e90-b08c-3100811d01be-kube-api-access-9m6dd\") pod \"dns-operator-controller-manager-7b84cd7c9c-27v92\" (UID: \"cf4cff2f-a2c3-4e90-b08c-3100811d01be\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-27v92"
Sep 18 18:47:16.657776 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:16.657748    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m6dd\" (UniqueName: \"kubernetes.io/projected/cf4cff2f-a2c3-4e90-b08c-3100811d01be-kube-api-access-9m6dd\") pod \"dns-operator-controller-manager-7b84cd7c9c-27v92\" (UID: \"cf4cff2f-a2c3-4e90-b08c-3100811d01be\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-27v92"
Sep 18 18:47:16.738468 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:16.738365    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-27v92"
Sep 18 18:47:17.023761 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:17.023737    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-27v92"]
Sep 18 18:47:17.025017 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:47:17.024992    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4cff2f_a2c3_4e90_b08c_3100811d01be.slice/crio-118d245cc98607092108e12f9564fe943b3451c1363b1d89eb5a20028ba8e248 WatchSource:0}: Error finding container 118d245cc98607092108e12f9564fe943b3451c1363b1d89eb5a20028ba8e248: Status 404 returned error can't find the container with id 118d245cc98607092108e12f9564fe943b3451c1363b1d89eb5a20028ba8e248
Sep 18 18:47:17.905809 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:17.905771    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-27v92" event={"ID":"cf4cff2f-a2c3-4e90-b08c-3100811d01be","Type":"ContainerStarted","Data":"118d245cc98607092108e12f9564fe943b3451c1363b1d89eb5a20028ba8e248"}
Sep 18 18:47:18.912435 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:18.912340    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-27v92" event={"ID":"cf4cff2f-a2c3-4e90-b08c-3100811d01be","Type":"ContainerStarted","Data":"776c16819dac079624a5e4ec75d4c7e8e0403600e4f2ea60b6a70e9d529c8a0b"}
Sep 18 18:47:18.912868 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:18.912459    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-27v92"
Sep 18 18:47:18.929313 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:18.929257    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-27v92" podStartSLOduration=1.325233627 podStartE2EDuration="2.929238283s" totalImagesPullingTime="1.604004656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:47:18.928437697 +0000 UTC m=+498.365016538" watchObservedRunningTime="2026-09-18 18:47:18.929238283 +0000 UTC m=+498.365817108"
Sep 18 18:47:20.980746 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:20.980711    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"]
Sep 18 18:47:20.984251 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:20.984230    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:20.986618 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:20.986599    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-99vf4\""
Sep 18 18:47:20.995079 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:20.995056    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"]
Sep 18 18:47:21.086837 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:21.086802    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqv69\" (UniqueName: \"kubernetes.io/projected/20f0e384-b0c7-4d96-af9f-dae08b793e7e-kube-api-access-sqv69\") pod \"kuadrant-operator-controller-manager-76f8f686bf-phv4b\" (UID: \"20f0e384-b0c7-4d96-af9f-dae08b793e7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:21.086837 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:21.086843    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20f0e384-b0c7-4d96-af9f-dae08b793e7e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-phv4b\" (UID: \"20f0e384-b0c7-4d96-af9f-dae08b793e7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:21.188095 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:21.188056    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqv69\" (UniqueName: \"kubernetes.io/projected/20f0e384-b0c7-4d96-af9f-dae08b793e7e-kube-api-access-sqv69\") pod \"kuadrant-operator-controller-manager-76f8f686bf-phv4b\" (UID: \"20f0e384-b0c7-4d96-af9f-dae08b793e7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:21.188095 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:21.188101    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20f0e384-b0c7-4d96-af9f-dae08b793e7e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-phv4b\" (UID: \"20f0e384-b0c7-4d96-af9f-dae08b793e7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:21.188473 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:21.188457    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20f0e384-b0c7-4d96-af9f-dae08b793e7e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-phv4b\" (UID: \"20f0e384-b0c7-4d96-af9f-dae08b793e7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:21.198262 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:21.198244    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqv69\" (UniqueName: \"kubernetes.io/projected/20f0e384-b0c7-4d96-af9f-dae08b793e7e-kube-api-access-sqv69\") pod \"kuadrant-operator-controller-manager-76f8f686bf-phv4b\" (UID: \"20f0e384-b0c7-4d96-af9f-dae08b793e7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:21.295367 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:21.295274    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:21.577005 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:21.576905    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"]
Sep 18 18:47:21.580484 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:47:21.580455    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f0e384_b0c7_4d96_af9f_dae08b793e7e.slice/crio-a75cd824d8d424b3e5c8b0996c8791d9d0dfa03d41a6cf8517d54ea695560033 WatchSource:0}: Error finding container a75cd824d8d424b3e5c8b0996c8791d9d0dfa03d41a6cf8517d54ea695560033: Status 404 returned error can't find the container with id a75cd824d8d424b3e5c8b0996c8791d9d0dfa03d41a6cf8517d54ea695560033
Sep 18 18:47:21.929224 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:21.929176    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b" event={"ID":"20f0e384-b0c7-4d96-af9f-dae08b793e7e","Type":"ContainerStarted","Data":"a75cd824d8d424b3e5c8b0996c8791d9d0dfa03d41a6cf8517d54ea695560033"}
Sep 18 18:47:25.896160 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:25.896127    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:25.957161 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:25.957126    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b" event={"ID":"20f0e384-b0c7-4d96-af9f-dae08b793e7e","Type":"ContainerStarted","Data":"0e340c564b1e6ac315669cd0d386b7160737f87efa583bc494469cbf37321c61"}
Sep 18 18:47:25.957350 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:25.957260    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:25.975150 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:25.975097    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b" podStartSLOduration=2.289160794 podStartE2EDuration="5.975080954s" totalImagesPullingTime="3.68592016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:47:25.973914304 +0000 UTC m=+505.410493146" watchObservedRunningTime="2026-09-18 18:47:25.975080954 +0000 UTC m=+505.411659795"
Sep 18 18:47:29.919246 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:29.919216    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-27v92"
Sep 18 18:47:36.965030 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:36.965000    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:38.977690 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:38.977655    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"]
Sep 18 18:47:38.978087 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:38.977849    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b" podUID="20f0e384-b0c7-4d96-af9f-dae08b793e7e" containerName="manager" containerID="cri-o://0e340c564b1e6ac315669cd0d386b7160737f87efa583bc494469cbf37321c61" gracePeriod=2
Sep 18 18:47:38.990607 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:38.990577    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"]
Sep 18 18:47:38.995139 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:38.995100    2558 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"20f0e384-b0c7-4d96-af9f-dae08b793e7e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-phv4b\": pods \"kuadrant-operator-controller-manager-76f8f686bf-phv4b\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:39.003857 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.003793    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"]
Sep 18 18:47:39.004143 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.004112    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4" podUID="d5d8a066-20d8-4861-b7a6-dff4f76210fa" containerName="manager" containerID="cri-o://ea686843382c30cbb5daa981c8fe86439e7891e2ad8466d8a8138a647414c0b1" gracePeriod=2
Sep 18 18:47:39.008031 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.008011    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"]
Sep 18 18:47:39.008610 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.008587    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f0e384-b0c7-4d96-af9f-dae08b793e7e" containerName="manager"
Sep 18 18:47:39.008610 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.008611    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f0e384-b0c7-4d96-af9f-dae08b793e7e" containerName="manager"
Sep 18 18:47:39.008772 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.008705    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="20f0e384-b0c7-4d96-af9f-dae08b793e7e" containerName="manager"
Sep 18 18:47:39.010285 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:39.010126    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-phv4b\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="20f0e384-b0c7-4d96-af9f-dae08b793e7e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:39.012364 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.012343    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"
Sep 18 18:47:39.012463 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.012394    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"]
Sep 18 18:47:39.015358 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:39.015330    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-phv4b\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="20f0e384-b0c7-4d96-af9f-dae08b793e7e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:39.029020 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.028992    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"]
Sep 18 18:47:39.036120 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.036094    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcjlp"]
Sep 18 18:47:39.036651 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.036631    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d8a066-20d8-4861-b7a6-dff4f76210fa" containerName="manager"
Sep 18 18:47:39.036698 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.036657    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d8a066-20d8-4861-b7a6-dff4f76210fa" containerName="manager"
Sep 18 18:47:39.036792 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.036779    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d8a066-20d8-4861-b7a6-dff4f76210fa" containerName="manager"
Sep 18 18:47:39.040267 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.040247    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcjlp"
Sep 18 18:47:39.041931 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.041908    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86689644-90c1-4471-8111-314cc31aa12d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7tcbl\" (UID: \"86689644-90c1-4471-8111-314cc31aa12d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"
Sep 18 18:47:39.042007 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.041950    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk844\" (UniqueName: \"kubernetes.io/projected/86689644-90c1-4471-8111-314cc31aa12d-kube-api-access-mk844\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7tcbl\" (UID: \"86689644-90c1-4471-8111-314cc31aa12d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"
Sep 18 18:47:39.042070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.042040    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmmbk\" (UniqueName: \"kubernetes.io/projected/ae6bba30-f072-4a01-83c0-80dacef1565b-kube-api-access-mmmbk\") pod \"limitador-operator-controller-manager-74495c69f8-mcjlp\" (UID: \"ae6bba30-f072-4a01-83c0-80dacef1565b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcjlp"
Sep 18 18:47:39.051353 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:39.051318    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xlzr4\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="d5d8a066-20d8-4861-b7a6-dff4f76210fa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:39.056835 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.056810    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcjlp"]
Sep 18 18:47:39.071529 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:39.071493    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xlzr4\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="d5d8a066-20d8-4861-b7a6-dff4f76210fa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:39.073297 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:39.073273    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-phv4b\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="20f0e384-b0c7-4d96-af9f-dae08b793e7e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:39.143112 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.143015    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86689644-90c1-4471-8111-314cc31aa12d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7tcbl\" (UID: \"86689644-90c1-4471-8111-314cc31aa12d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"
Sep 18 18:47:39.143112 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.143080    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk844\" (UniqueName: \"kubernetes.io/projected/86689644-90c1-4471-8111-314cc31aa12d-kube-api-access-mk844\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7tcbl\" (UID: \"86689644-90c1-4471-8111-314cc31aa12d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"
Sep 18 18:47:39.143400 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.143131    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmmbk\" (UniqueName: \"kubernetes.io/projected/ae6bba30-f072-4a01-83c0-80dacef1565b-kube-api-access-mmmbk\") pod \"limitador-operator-controller-manager-74495c69f8-mcjlp\" (UID: \"ae6bba30-f072-4a01-83c0-80dacef1565b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcjlp"
Sep 18 18:47:39.143949 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.143897    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86689644-90c1-4471-8111-314cc31aa12d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7tcbl\" (UID: \"86689644-90c1-4471-8111-314cc31aa12d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"
Sep 18 18:47:39.158845 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.158816    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk844\" (UniqueName: \"kubernetes.io/projected/86689644-90c1-4471-8111-314cc31aa12d-kube-api-access-mk844\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7tcbl\" (UID: \"86689644-90c1-4471-8111-314cc31aa12d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"
Sep 18 18:47:39.158961 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.158859    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmmbk\" (UniqueName: \"kubernetes.io/projected/ae6bba30-f072-4a01-83c0-80dacef1565b-kube-api-access-mmmbk\") pod \"limitador-operator-controller-manager-74495c69f8-mcjlp\" (UID: \"ae6bba30-f072-4a01-83c0-80dacef1565b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcjlp"
Sep 18 18:47:39.260177 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.260154    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:39.262319 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.262299    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:39.262416 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:39.262295    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xlzr4\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="d5d8a066-20d8-4861-b7a6-dff4f76210fa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:39.264114 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:39.264092    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-phv4b\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="20f0e384-b0c7-4d96-af9f-dae08b793e7e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:39.266006 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:39.265987    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-phv4b\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="20f0e384-b0c7-4d96-af9f-dae08b793e7e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:39.267572 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:39.267554    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xlzr4\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="d5d8a066-20d8-4861-b7a6-dff4f76210fa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:39.344876 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.344845    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20f0e384-b0c7-4d96-af9f-dae08b793e7e-extensions-socket-volume\") pod \"20f0e384-b0c7-4d96-af9f-dae08b793e7e\" (UID: \"20f0e384-b0c7-4d96-af9f-dae08b793e7e\") "
Sep 18 18:47:39.344876 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.344881    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqv69\" (UniqueName: \"kubernetes.io/projected/20f0e384-b0c7-4d96-af9f-dae08b793e7e-kube-api-access-sqv69\") pod \"20f0e384-b0c7-4d96-af9f-dae08b793e7e\" (UID: \"20f0e384-b0c7-4d96-af9f-dae08b793e7e\") "
Sep 18 18:47:39.345092 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.344926    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crd6c\" (UniqueName: \"kubernetes.io/projected/d5d8a066-20d8-4861-b7a6-dff4f76210fa-kube-api-access-crd6c\") pod \"d5d8a066-20d8-4861-b7a6-dff4f76210fa\" (UID: \"d5d8a066-20d8-4861-b7a6-dff4f76210fa\") "
Sep 18 18:47:39.345349 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.345325    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f0e384-b0c7-4d96-af9f-dae08b793e7e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "20f0e384-b0c7-4d96-af9f-dae08b793e7e" (UID: "20f0e384-b0c7-4d96-af9f-dae08b793e7e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:47:39.347140 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.347118    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d8a066-20d8-4861-b7a6-dff4f76210fa-kube-api-access-crd6c" (OuterVolumeSpecName: "kube-api-access-crd6c") pod "d5d8a066-20d8-4861-b7a6-dff4f76210fa" (UID: "d5d8a066-20d8-4861-b7a6-dff4f76210fa"). InnerVolumeSpecName "kube-api-access-crd6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:47:39.347219 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.347162    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f0e384-b0c7-4d96-af9f-dae08b793e7e-kube-api-access-sqv69" (OuterVolumeSpecName: "kube-api-access-sqv69") pod "20f0e384-b0c7-4d96-af9f-dae08b793e7e" (UID: "20f0e384-b0c7-4d96-af9f-dae08b793e7e"). InnerVolumeSpecName "kube-api-access-sqv69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:47:39.410978 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.410940    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"
Sep 18 18:47:39.417960 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.417941    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcjlp"
Sep 18 18:47:39.445528 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.445499    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20f0e384-b0c7-4d96-af9f-dae08b793e7e-extensions-socket-volume\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:39.445528 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.445526    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqv69\" (UniqueName: \"kubernetes.io/projected/20f0e384-b0c7-4d96-af9f-dae08b793e7e-kube-api-access-sqv69\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:39.445703 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.445539    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crd6c\" (UniqueName: \"kubernetes.io/projected/d5d8a066-20d8-4861-b7a6-dff4f76210fa-kube-api-access-crd6c\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:39.694261 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.694238    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"]
Sep 18 18:47:39.695236 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:47:39.695209    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86689644_90c1_4471_8111_314cc31aa12d.slice/crio-0eea4a4267effad876bf86d5d841b08f72edac02df9e657ad1e6fb12731e37de WatchSource:0}: Error finding container 0eea4a4267effad876bf86d5d841b08f72edac02df9e657ad1e6fb12731e37de: Status 404 returned error can't find the container with id 0eea4a4267effad876bf86d5d841b08f72edac02df9e657ad1e6fb12731e37de
Sep 18 18:47:39.703829 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:39.703805    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcjlp"]
Sep 18 18:47:39.709029 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:47:39.709007    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6bba30_f072_4a01_83c0_80dacef1565b.slice/crio-952a3999d00b02d639ab84a755e5336b18568647bd10e688073577b5c06b8afe WatchSource:0}: Error finding container 952a3999d00b02d639ab84a755e5336b18568647bd10e688073577b5c06b8afe: Status 404 returned error can't find the container with id 952a3999d00b02d639ab84a755e5336b18568647bd10e688073577b5c06b8afe
Sep 18 18:47:40.039426 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.039388    2558 generic.go:356] "Generic (PLEG): container finished" podID="d5d8a066-20d8-4861-b7a6-dff4f76210fa" containerID="ea686843382c30cbb5daa981c8fe86439e7891e2ad8466d8a8138a647414c0b1" exitCode=0
Sep 18 18:47:40.039892 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.039444    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:40.039892 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.039507    2558 scope.go:117] "RemoveContainer" containerID="ea686843382c30cbb5daa981c8fe86439e7891e2ad8466d8a8138a647414c0b1"
Sep 18 18:47:40.041495 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:40.041464    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xlzr4\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="d5d8a066-20d8-4861-b7a6-dff4f76210fa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:40.041644 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.041540    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl" event={"ID":"86689644-90c1-4471-8111-314cc31aa12d","Type":"ContainerStarted","Data":"20aea3323253b876d0089c6cdf625214406251c5911e12822fee8c45e5227b3c"}
Sep 18 18:47:40.041644 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.041595    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"
Sep 18 18:47:40.041644 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.041620    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl" event={"ID":"86689644-90c1-4471-8111-314cc31aa12d","Type":"ContainerStarted","Data":"0eea4a4267effad876bf86d5d841b08f72edac02df9e657ad1e6fb12731e37de"}
Sep 18 18:47:40.043129 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:40.043104    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-phv4b\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="20f0e384-b0c7-4d96-af9f-dae08b793e7e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:40.043431 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.043402    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcjlp" event={"ID":"ae6bba30-f072-4a01-83c0-80dacef1565b","Type":"ContainerStarted","Data":"1b23ddef2eef702be08dc4de8cb1893d62ddf9f8b25f73bc2957933867730c31"}
Sep 18 18:47:40.043521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.043448    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcjlp" event={"ID":"ae6bba30-f072-4a01-83c0-80dacef1565b","Type":"ContainerStarted","Data":"952a3999d00b02d639ab84a755e5336b18568647bd10e688073577b5c06b8afe"}
Sep 18 18:47:40.043578 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.043520    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcjlp"
Sep 18 18:47:40.044957 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.044934    2558 generic.go:356] "Generic (PLEG): container finished" podID="20f0e384-b0c7-4d96-af9f-dae08b793e7e" containerID="0e340c564b1e6ac315669cd0d386b7160737f87efa583bc494469cbf37321c61" exitCode=0
Sep 18 18:47:40.045046 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.044976    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:40.050707 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.050670    2558 scope.go:117] "RemoveContainer" containerID="ea686843382c30cbb5daa981c8fe86439e7891e2ad8466d8a8138a647414c0b1"
Sep 18 18:47:40.051090 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:40.051047    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea686843382c30cbb5daa981c8fe86439e7891e2ad8466d8a8138a647414c0b1\": container with ID starting with ea686843382c30cbb5daa981c8fe86439e7891e2ad8466d8a8138a647414c0b1 not found: ID does not exist" containerID="ea686843382c30cbb5daa981c8fe86439e7891e2ad8466d8a8138a647414c0b1"
Sep 18 18:47:40.051230 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.051100    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea686843382c30cbb5daa981c8fe86439e7891e2ad8466d8a8138a647414c0b1"} err="failed to get container status \"ea686843382c30cbb5daa981c8fe86439e7891e2ad8466d8a8138a647414c0b1\": rpc error: code = NotFound desc = could not find container \"ea686843382c30cbb5daa981c8fe86439e7891e2ad8466d8a8138a647414c0b1\": container with ID starting with ea686843382c30cbb5daa981c8fe86439e7891e2ad8466d8a8138a647414c0b1 not found: ID does not exist"
Sep 18 18:47:40.051230 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.051130    2558 scope.go:117] "RemoveContainer" containerID="0e340c564b1e6ac315669cd0d386b7160737f87efa583bc494469cbf37321c61"
Sep 18 18:47:40.060530 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.060513    2558 scope.go:117] "RemoveContainer" containerID="0e340c564b1e6ac315669cd0d386b7160737f87efa583bc494469cbf37321c61"
Sep 18 18:47:40.060795 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:40.060778    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e340c564b1e6ac315669cd0d386b7160737f87efa583bc494469cbf37321c61\": container with ID starting with 0e340c564b1e6ac315669cd0d386b7160737f87efa583bc494469cbf37321c61 not found: ID does not exist" containerID="0e340c564b1e6ac315669cd0d386b7160737f87efa583bc494469cbf37321c61"
Sep 18 18:47:40.060857 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.060802    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e340c564b1e6ac315669cd0d386b7160737f87efa583bc494469cbf37321c61"} err="failed to get container status \"0e340c564b1e6ac315669cd0d386b7160737f87efa583bc494469cbf37321c61\": rpc error: code = NotFound desc = could not find container \"0e340c564b1e6ac315669cd0d386b7160737f87efa583bc494469cbf37321c61\": container with ID starting with 0e340c564b1e6ac315669cd0d386b7160737f87efa583bc494469cbf37321c61 not found: ID does not exist"
Sep 18 18:47:40.065734 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.065694    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl" podStartSLOduration=2.065679229 podStartE2EDuration="2.065679229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:47:40.06545429 +0000 UTC m=+519.502033133" watchObservedRunningTime="2026-09-18 18:47:40.065679229 +0000 UTC m=+519.502258072"
Sep 18 18:47:40.067577 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:40.067551    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xlzr4\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="d5d8a066-20d8-4861-b7a6-dff4f76210fa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:40.069327 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:40.069297    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-phv4b\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="20f0e384-b0c7-4d96-af9f-dae08b793e7e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:40.070997 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:40.070962    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-phv4b\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="20f0e384-b0c7-4d96-af9f-dae08b793e7e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-phv4b"
Sep 18 18:47:40.090847 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.090801    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcjlp" podStartSLOduration=1.090787487 podStartE2EDuration="1.090787487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:47:40.089311229 +0000 UTC m=+519.525890062" watchObservedRunningTime="2026-09-18 18:47:40.090787487 +0000 UTC m=+519.527366327"
Sep 18 18:47:40.091582 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:40.091559    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xlzr4\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="d5d8a066-20d8-4861-b7a6-dff4f76210fa" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xlzr4"
Sep 18 18:47:40.576713 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.576679    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20f0e384-b0c7-4d96-af9f-dae08b793e7e" path="/var/lib/kubelet/pods/20f0e384-b0c7-4d96-af9f-dae08b793e7e/volumes"
Sep 18 18:47:40.577004 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:40.576991    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5d8a066-20d8-4861-b7a6-dff4f76210fa" path="/var/lib/kubelet/pods/d5d8a066-20d8-4861-b7a6-dff4f76210fa/volumes"
Sep 18 18:47:51.054907 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:51.054832    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"
Sep 18 18:47:51.055410 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:51.054918    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mcjlp"
Sep 18 18:47:55.189474 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.189433    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"]
Sep 18 18:47:55.189911 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.189672    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl" podUID="86689644-90c1-4471-8111-314cc31aa12d" containerName="manager" containerID="cri-o://20aea3323253b876d0089c6cdf625214406251c5911e12822fee8c45e5227b3c" gracePeriod=10
Sep 18 18:47:55.390474 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.390440    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"]
Sep 18 18:47:55.395296 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.395268    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"
Sep 18 18:47:55.408499 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.408471    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"]
Sep 18 18:47:55.444320 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.444267    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"
Sep 18 18:47:55.485106 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.485077    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk844\" (UniqueName: \"kubernetes.io/projected/86689644-90c1-4471-8111-314cc31aa12d-kube-api-access-mk844\") pod \"86689644-90c1-4471-8111-314cc31aa12d\" (UID: \"86689644-90c1-4471-8111-314cc31aa12d\") "
Sep 18 18:47:55.485287 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.485126    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86689644-90c1-4471-8111-314cc31aa12d-extensions-socket-volume\") pod \"86689644-90c1-4471-8111-314cc31aa12d\" (UID: \"86689644-90c1-4471-8111-314cc31aa12d\") "
Sep 18 18:47:55.485387 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.485368    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqq5b\" (UniqueName: \"kubernetes.io/projected/792ae493-6c54-472b-bd71-f144fd3f28e3-kube-api-access-jqq5b\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tzr6p\" (UID: \"792ae493-6c54-472b-bd71-f144fd3f28e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"
Sep 18 18:47:55.485455 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.485406    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/792ae493-6c54-472b-bd71-f144fd3f28e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tzr6p\" (UID: \"792ae493-6c54-472b-bd71-f144fd3f28e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"
Sep 18 18:47:55.485629 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.485603    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86689644-90c1-4471-8111-314cc31aa12d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "86689644-90c1-4471-8111-314cc31aa12d" (UID: "86689644-90c1-4471-8111-314cc31aa12d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:47:55.487310 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.487279    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86689644-90c1-4471-8111-314cc31aa12d-kube-api-access-mk844" (OuterVolumeSpecName: "kube-api-access-mk844") pod "86689644-90c1-4471-8111-314cc31aa12d" (UID: "86689644-90c1-4471-8111-314cc31aa12d"). InnerVolumeSpecName "kube-api-access-mk844". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:47:55.586738 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.586702    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqq5b\" (UniqueName: \"kubernetes.io/projected/792ae493-6c54-472b-bd71-f144fd3f28e3-kube-api-access-jqq5b\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tzr6p\" (UID: \"792ae493-6c54-472b-bd71-f144fd3f28e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"
Sep 18 18:47:55.586925 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.586744    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/792ae493-6c54-472b-bd71-f144fd3f28e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tzr6p\" (UID: \"792ae493-6c54-472b-bd71-f144fd3f28e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"
Sep 18 18:47:55.586925 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.586787    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk844\" (UniqueName: \"kubernetes.io/projected/86689644-90c1-4471-8111-314cc31aa12d-kube-api-access-mk844\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:55.586925 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.586798    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86689644-90c1-4471-8111-314cc31aa12d-extensions-socket-volume\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:47:55.587116 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.587101    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/792ae493-6c54-472b-bd71-f144fd3f28e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tzr6p\" (UID: \"792ae493-6c54-472b-bd71-f144fd3f28e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"
Sep 18 18:47:55.595359 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.595336    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqq5b\" (UniqueName: \"kubernetes.io/projected/792ae493-6c54-472b-bd71-f144fd3f28e3-kube-api-access-jqq5b\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tzr6p\" (UID: \"792ae493-6c54-472b-bd71-f144fd3f28e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"
Sep 18 18:47:55.705689 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.705594    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"
Sep 18 18:47:55.987571 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:55.987549    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"]
Sep 18 18:47:55.989522 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:47:55.989494    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792ae493_6c54_472b_bd71_f144fd3f28e3.slice/crio-b46c7e73760f0654c68af85dd6aa3a6c6f41846f75a2eb2245980ed905875b8e WatchSource:0}: Error finding container b46c7e73760f0654c68af85dd6aa3a6c6f41846f75a2eb2245980ed905875b8e: Status 404 returned error can't find the container with id b46c7e73760f0654c68af85dd6aa3a6c6f41846f75a2eb2245980ed905875b8e
Sep 18 18:47:56.138109 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:56.138069    2558 generic.go:356] "Generic (PLEG): container finished" podID="86689644-90c1-4471-8111-314cc31aa12d" containerID="20aea3323253b876d0089c6cdf625214406251c5911e12822fee8c45e5227b3c" exitCode=0
Sep 18 18:47:56.138312 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:56.138124    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"
Sep 18 18:47:56.138312 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:56.138153    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl" event={"ID":"86689644-90c1-4471-8111-314cc31aa12d","Type":"ContainerDied","Data":"20aea3323253b876d0089c6cdf625214406251c5911e12822fee8c45e5227b3c"}
Sep 18 18:47:56.138312 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:56.138243    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl" event={"ID":"86689644-90c1-4471-8111-314cc31aa12d","Type":"ContainerDied","Data":"0eea4a4267effad876bf86d5d841b08f72edac02df9e657ad1e6fb12731e37de"}
Sep 18 18:47:56.138312 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:56.138282    2558 scope.go:117] "RemoveContainer" containerID="20aea3323253b876d0089c6cdf625214406251c5911e12822fee8c45e5227b3c"
Sep 18 18:47:56.140240 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:56.140221    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p" event={"ID":"792ae493-6c54-472b-bd71-f144fd3f28e3","Type":"ContainerStarted","Data":"5c7682f89f129ec04bbad57ebbb4c577a25bf493d74eedcb48ae1db9a888d794"}
Sep 18 18:47:56.140354 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:56.140261    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p" event={"ID":"792ae493-6c54-472b-bd71-f144fd3f28e3","Type":"ContainerStarted","Data":"b46c7e73760f0654c68af85dd6aa3a6c6f41846f75a2eb2245980ed905875b8e"}
Sep 18 18:47:56.140417 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:56.140381    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"
Sep 18 18:47:56.148168 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:56.148147    2558 scope.go:117] "RemoveContainer" containerID="20aea3323253b876d0089c6cdf625214406251c5911e12822fee8c45e5227b3c"
Sep 18 18:47:56.148480 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:47:56.148460    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20aea3323253b876d0089c6cdf625214406251c5911e12822fee8c45e5227b3c\": container with ID starting with 20aea3323253b876d0089c6cdf625214406251c5911e12822fee8c45e5227b3c not found: ID does not exist" containerID="20aea3323253b876d0089c6cdf625214406251c5911e12822fee8c45e5227b3c"
Sep 18 18:47:56.148558 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:56.148493    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20aea3323253b876d0089c6cdf625214406251c5911e12822fee8c45e5227b3c"} err="failed to get container status \"20aea3323253b876d0089c6cdf625214406251c5911e12822fee8c45e5227b3c\": rpc error: code = NotFound desc = could not find container \"20aea3323253b876d0089c6cdf625214406251c5911e12822fee8c45e5227b3c\": container with ID starting with 20aea3323253b876d0089c6cdf625214406251c5911e12822fee8c45e5227b3c not found: ID does not exist"
Sep 18 18:47:56.162570 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:56.162516    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p" podStartSLOduration=1.162498294 podStartE2EDuration="1.162498294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:47:56.161963244 +0000 UTC m=+535.598542123" watchObservedRunningTime="2026-09-18 18:47:56.162498294 +0000 UTC m=+535.599077151"
Sep 18 18:47:56.180580 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:56.180552    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"]
Sep 18 18:47:56.184493 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:56.184468    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7tcbl"]
Sep 18 18:47:56.575704 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:47:56.575667    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86689644-90c1-4471-8111-314cc31aa12d" path="/var/lib/kubelet/pods/86689644-90c1-4471-8111-314cc31aa12d/volumes"
Sep 18 18:48:07.150580 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:07.150549    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"
Sep 18 18:48:25.402180 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:25.402145    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-f7qfl"]
Sep 18 18:48:25.402915 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:25.402520    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86689644-90c1-4471-8111-314cc31aa12d" containerName="manager"
Sep 18 18:48:25.402915 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:25.402534    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="86689644-90c1-4471-8111-314cc31aa12d" containerName="manager"
Sep 18 18:48:25.402915 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:25.402603    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="86689644-90c1-4471-8111-314cc31aa12d" containerName="manager"
Sep 18 18:48:25.405492 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:25.405474    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-f7qfl"
Sep 18 18:48:25.407879 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:25.407858    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6xsp7\""
Sep 18 18:48:25.410785 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:25.410762    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-f7qfl"]
Sep 18 18:48:25.543470 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:25.543431    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwn7\" (UniqueName: \"kubernetes.io/projected/7ec64955-89aa-41df-9d6e-696cf3cc7a0c-kube-api-access-hzwn7\") pod \"authorino-7bcfd8f9cd-f7qfl\" (UID: \"7ec64955-89aa-41df-9d6e-696cf3cc7a0c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-f7qfl"
Sep 18 18:48:25.644107 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:25.644068    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwn7\" (UniqueName: \"kubernetes.io/projected/7ec64955-89aa-41df-9d6e-696cf3cc7a0c-kube-api-access-hzwn7\") pod \"authorino-7bcfd8f9cd-f7qfl\" (UID: \"7ec64955-89aa-41df-9d6e-696cf3cc7a0c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-f7qfl"
Sep 18 18:48:25.652750 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:25.652693    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwn7\" (UniqueName: \"kubernetes.io/projected/7ec64955-89aa-41df-9d6e-696cf3cc7a0c-kube-api-access-hzwn7\") pod \"authorino-7bcfd8f9cd-f7qfl\" (UID: \"7ec64955-89aa-41df-9d6e-696cf3cc7a0c\") " pod="kuadrant-system/authorino-7bcfd8f9cd-f7qfl"
Sep 18 18:48:25.715801 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:25.715767    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-f7qfl"
Sep 18 18:48:25.987044 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:25.987013    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-f7qfl"]
Sep 18 18:48:25.989024 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:48:25.988999    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ec64955_89aa_41df_9d6e_696cf3cc7a0c.slice/crio-10eba15c9299e0b3bacf952f0b929f8ff072b4cf48b228e67d27876b90edb8f4 WatchSource:0}: Error finding container 10eba15c9299e0b3bacf952f0b929f8ff072b4cf48b228e67d27876b90edb8f4: Status 404 returned error can't find the container with id 10eba15c9299e0b3bacf952f0b929f8ff072b4cf48b228e67d27876b90edb8f4
Sep 18 18:48:26.313692 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:26.313605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-f7qfl" event={"ID":"7ec64955-89aa-41df-9d6e-696cf3cc7a0c","Type":"ContainerStarted","Data":"10eba15c9299e0b3bacf952f0b929f8ff072b4cf48b228e67d27876b90edb8f4"}
Sep 18 18:48:28.333247 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:28.333130    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-f7qfl" event={"ID":"7ec64955-89aa-41df-9d6e-696cf3cc7a0c","Type":"ContainerStarted","Data":"8fc41f6909d7d9d171820f894ad138ccdde37b3ea336c1d195520262fe8cf069"}
Sep 18 18:48:28.348176 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:28.348129    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-f7qfl" podStartSLOduration=1.358350011 podStartE2EDuration="3.34810429s" totalImagesPullingTime="1.989754279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:48:28.347352934 +0000 UTC m=+567.783931788" watchObservedRunningTime="2026-09-18 18:48:28.34810429 +0000 UTC m=+567.784683133"
Sep 18 18:48:29.404959 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:29.404921    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-f7qfl"]
Sep 18 18:48:30.343679 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:30.343622    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-f7qfl" podUID="7ec64955-89aa-41df-9d6e-696cf3cc7a0c" containerName="authorino" containerID="cri-o://8fc41f6909d7d9d171820f894ad138ccdde37b3ea336c1d195520262fe8cf069" gracePeriod=30
Sep 18 18:48:30.591350 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:30.591323    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-f7qfl"
Sep 18 18:48:30.688155 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:30.688049    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzwn7\" (UniqueName: \"kubernetes.io/projected/7ec64955-89aa-41df-9d6e-696cf3cc7a0c-kube-api-access-hzwn7\") pod \"7ec64955-89aa-41df-9d6e-696cf3cc7a0c\" (UID: \"7ec64955-89aa-41df-9d6e-696cf3cc7a0c\") "
Sep 18 18:48:30.690154 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:30.690125    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec64955-89aa-41df-9d6e-696cf3cc7a0c-kube-api-access-hzwn7" (OuterVolumeSpecName: "kube-api-access-hzwn7") pod "7ec64955-89aa-41df-9d6e-696cf3cc7a0c" (UID: "7ec64955-89aa-41df-9d6e-696cf3cc7a0c"). InnerVolumeSpecName "kube-api-access-hzwn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:48:30.789316 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:30.789284    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzwn7\" (UniqueName: \"kubernetes.io/projected/7ec64955-89aa-41df-9d6e-696cf3cc7a0c-kube-api-access-hzwn7\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:48:31.350396 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:31.350362    2558 generic.go:356] "Generic (PLEG): container finished" podID="7ec64955-89aa-41df-9d6e-696cf3cc7a0c" containerID="8fc41f6909d7d9d171820f894ad138ccdde37b3ea336c1d195520262fe8cf069" exitCode=0
Sep 18 18:48:31.350572 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:31.350406    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-f7qfl"
Sep 18 18:48:31.350572 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:31.350452    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-f7qfl" event={"ID":"7ec64955-89aa-41df-9d6e-696cf3cc7a0c","Type":"ContainerDied","Data":"8fc41f6909d7d9d171820f894ad138ccdde37b3ea336c1d195520262fe8cf069"}
Sep 18 18:48:31.350572 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:31.350493    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-f7qfl" event={"ID":"7ec64955-89aa-41df-9d6e-696cf3cc7a0c","Type":"ContainerDied","Data":"10eba15c9299e0b3bacf952f0b929f8ff072b4cf48b228e67d27876b90edb8f4"}
Sep 18 18:48:31.350572 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:31.350512    2558 scope.go:117] "RemoveContainer" containerID="8fc41f6909d7d9d171820f894ad138ccdde37b3ea336c1d195520262fe8cf069"
Sep 18 18:48:31.359798 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:31.359779    2558 scope.go:117] "RemoveContainer" containerID="8fc41f6909d7d9d171820f894ad138ccdde37b3ea336c1d195520262fe8cf069"
Sep 18 18:48:31.360055 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:48:31.360036    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc41f6909d7d9d171820f894ad138ccdde37b3ea336c1d195520262fe8cf069\": container with ID starting with 8fc41f6909d7d9d171820f894ad138ccdde37b3ea336c1d195520262fe8cf069 not found: ID does not exist" containerID="8fc41f6909d7d9d171820f894ad138ccdde37b3ea336c1d195520262fe8cf069"
Sep 18 18:48:31.360108 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:31.360063    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc41f6909d7d9d171820f894ad138ccdde37b3ea336c1d195520262fe8cf069"} err="failed to get container status \"8fc41f6909d7d9d171820f894ad138ccdde37b3ea336c1d195520262fe8cf069\": rpc error: code = NotFound desc = could not find container \"8fc41f6909d7d9d171820f894ad138ccdde37b3ea336c1d195520262fe8cf069\": container with ID starting with 8fc41f6909d7d9d171820f894ad138ccdde37b3ea336c1d195520262fe8cf069 not found: ID does not exist"
Sep 18 18:48:31.370844 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:31.370819    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-f7qfl"]
Sep 18 18:48:31.374600 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:31.374580    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-f7qfl"]
Sep 18 18:48:32.576325 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:32.576294    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ec64955-89aa-41df-9d6e-696cf3cc7a0c" path="/var/lib/kubelet/pods/7ec64955-89aa-41df-9d6e-696cf3cc7a0c/volumes"
Sep 18 18:48:56.010864 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.010827    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-6w4b9"]
Sep 18 18:48:56.011268 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.011207    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec64955-89aa-41df-9d6e-696cf3cc7a0c" containerName="authorino"
Sep 18 18:48:56.011268 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.011219    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec64955-89aa-41df-9d6e-696cf3cc7a0c" containerName="authorino"
Sep 18 18:48:56.011357 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.011284    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ec64955-89aa-41df-9d6e-696cf3cc7a0c" containerName="authorino"
Sep 18 18:48:56.014320 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.014304    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6w4b9"
Sep 18 18:48:56.016480 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.016460    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6xsp7\""
Sep 18 18:48:56.020364 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.020334    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-6w4b9"]
Sep 18 18:48:56.107464 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.107431    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzsrt\" (UniqueName: \"kubernetes.io/projected/ddab2017-6c34-41fd-b7b6-496228df51ce-kube-api-access-tzsrt\") pod \"authorino-557f876f69-6w4b9\" (UID: \"ddab2017-6c34-41fd-b7b6-496228df51ce\") " pod="kuadrant-system/authorino-557f876f69-6w4b9"
Sep 18 18:48:56.149069 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.149038    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6w4b9"]
Sep 18 18:48:56.149327 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:48:56.149306    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tzsrt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-6w4b9" podUID="ddab2017-6c34-41fd-b7b6-496228df51ce"
Sep 18 18:48:56.175862 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.175823    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-8l829"]
Sep 18 18:48:56.179255 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.179234    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-8l829"
Sep 18 18:48:56.181239 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.181221    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 18:48:56.185755 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.185736    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-8l829"]
Sep 18 18:48:56.208528 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.208498    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzsrt\" (UniqueName: \"kubernetes.io/projected/ddab2017-6c34-41fd-b7b6-496228df51ce-kube-api-access-tzsrt\") pod \"authorino-557f876f69-6w4b9\" (UID: \"ddab2017-6c34-41fd-b7b6-496228df51ce\") " pod="kuadrant-system/authorino-557f876f69-6w4b9"
Sep 18 18:48:56.218979 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.218956    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzsrt\" (UniqueName: \"kubernetes.io/projected/ddab2017-6c34-41fd-b7b6-496228df51ce-kube-api-access-tzsrt\") pod \"authorino-557f876f69-6w4b9\" (UID: \"ddab2017-6c34-41fd-b7b6-496228df51ce\") " pod="kuadrant-system/authorino-557f876f69-6w4b9"
Sep 18 18:48:56.230263 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.230241    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-8l829"]
Sep 18 18:48:56.230470 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:48:56.230451    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-g26c6 tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-g26c6 tls-cert]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-8l829" podUID="81e5b54a-ff00-4c71-bedc-a8e07193cf69"
Sep 18 18:48:56.260344 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.260318    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fd94687d8-v9xrv"]
Sep 18 18:48:56.264311 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.264258    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fd94687d8-v9xrv"
Sep 18 18:48:56.274639 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.274618    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fd94687d8-v9xrv"]
Sep 18 18:48:56.309486 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.309451    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81e5b54a-ff00-4c71-bedc-a8e07193cf69-tls-cert\") pod \"authorino-6d4dfc649f-8l829\" (UID: \"81e5b54a-ff00-4c71-bedc-a8e07193cf69\") " pod="kuadrant-system/authorino-6d4dfc649f-8l829"
Sep 18 18:48:56.309701 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.309512    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26c6\" (UniqueName: \"kubernetes.io/projected/81e5b54a-ff00-4c71-bedc-a8e07193cf69-kube-api-access-g26c6\") pod \"authorino-6d4dfc649f-8l829\" (UID: \"81e5b54a-ff00-4c71-bedc-a8e07193cf69\") " pod="kuadrant-system/authorino-6d4dfc649f-8l829"
Sep 18 18:48:56.410783 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.410749    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81e5b54a-ff00-4c71-bedc-a8e07193cf69-tls-cert\") pod \"authorino-6d4dfc649f-8l829\" (UID: \"81e5b54a-ff00-4c71-bedc-a8e07193cf69\") " pod="kuadrant-system/authorino-6d4dfc649f-8l829"
Sep 18 18:48:56.410953 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.410803    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83ed3401-aac9-48a9-9b6f-211e9db35d3e-tls-cert\") pod \"authorino-5fd94687d8-v9xrv\" (UID: \"83ed3401-aac9-48a9-9b6f-211e9db35d3e\") " pod="kuadrant-system/authorino-5fd94687d8-v9xrv"
Sep 18 18:48:56.410953 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.410865    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89cr\" (UniqueName: \"kubernetes.io/projected/83ed3401-aac9-48a9-9b6f-211e9db35d3e-kube-api-access-v89cr\") pod \"authorino-5fd94687d8-v9xrv\" (UID: \"83ed3401-aac9-48a9-9b6f-211e9db35d3e\") " pod="kuadrant-system/authorino-5fd94687d8-v9xrv"
Sep 18 18:48:56.410953 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.410888    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26c6\" (UniqueName: \"kubernetes.io/projected/81e5b54a-ff00-4c71-bedc-a8e07193cf69-kube-api-access-g26c6\") pod \"authorino-6d4dfc649f-8l829\" (UID: \"81e5b54a-ff00-4c71-bedc-a8e07193cf69\") " pod="kuadrant-system/authorino-6d4dfc649f-8l829"
Sep 18 18:48:56.413226 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.413199    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81e5b54a-ff00-4c71-bedc-a8e07193cf69-tls-cert\") pod \"authorino-6d4dfc649f-8l829\" (UID: \"81e5b54a-ff00-4c71-bedc-a8e07193cf69\") " pod="kuadrant-system/authorino-6d4dfc649f-8l829"
Sep 18 18:48:56.418723 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.418701    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26c6\" (UniqueName: \"kubernetes.io/projected/81e5b54a-ff00-4c71-bedc-a8e07193cf69-kube-api-access-g26c6\") pod \"authorino-6d4dfc649f-8l829\" (UID: \"81e5b54a-ff00-4c71-bedc-a8e07193cf69\") " pod="kuadrant-system/authorino-6d4dfc649f-8l829"
Sep 18 18:48:56.487508 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.487478    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-8l829"
Sep 18 18:48:56.487653 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.487482    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6w4b9"
Sep 18 18:48:56.494824 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.494803    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-8l829"
Sep 18 18:48:56.497620 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.497604    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6w4b9"
Sep 18 18:48:56.511589 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.511566    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89cr\" (UniqueName: \"kubernetes.io/projected/83ed3401-aac9-48a9-9b6f-211e9db35d3e-kube-api-access-v89cr\") pod \"authorino-5fd94687d8-v9xrv\" (UID: \"83ed3401-aac9-48a9-9b6f-211e9db35d3e\") " pod="kuadrant-system/authorino-5fd94687d8-v9xrv"
Sep 18 18:48:56.511704 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.511659    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83ed3401-aac9-48a9-9b6f-211e9db35d3e-tls-cert\") pod \"authorino-5fd94687d8-v9xrv\" (UID: \"83ed3401-aac9-48a9-9b6f-211e9db35d3e\") " pod="kuadrant-system/authorino-5fd94687d8-v9xrv"
Sep 18 18:48:56.514141 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.514113    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83ed3401-aac9-48a9-9b6f-211e9db35d3e-tls-cert\") pod \"authorino-5fd94687d8-v9xrv\" (UID: \"83ed3401-aac9-48a9-9b6f-211e9db35d3e\") " pod="kuadrant-system/authorino-5fd94687d8-v9xrv"
Sep 18 18:48:56.519087 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.519038    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89cr\" (UniqueName: \"kubernetes.io/projected/83ed3401-aac9-48a9-9b6f-211e9db35d3e-kube-api-access-v89cr\") pod \"authorino-5fd94687d8-v9xrv\" (UID: \"83ed3401-aac9-48a9-9b6f-211e9db35d3e\") " pod="kuadrant-system/authorino-5fd94687d8-v9xrv"
Sep 18 18:48:56.574164 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.574142    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fd94687d8-v9xrv"
Sep 18 18:48:56.612761 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.612732    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzsrt\" (UniqueName: \"kubernetes.io/projected/ddab2017-6c34-41fd-b7b6-496228df51ce-kube-api-access-tzsrt\") pod \"ddab2017-6c34-41fd-b7b6-496228df51ce\" (UID: \"ddab2017-6c34-41fd-b7b6-496228df51ce\") "
Sep 18 18:48:56.613943 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.612830    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g26c6\" (UniqueName: \"kubernetes.io/projected/81e5b54a-ff00-4c71-bedc-a8e07193cf69-kube-api-access-g26c6\") pod \"81e5b54a-ff00-4c71-bedc-a8e07193cf69\" (UID: \"81e5b54a-ff00-4c71-bedc-a8e07193cf69\") "
Sep 18 18:48:56.613943 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.612862    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81e5b54a-ff00-4c71-bedc-a8e07193cf69-tls-cert\") pod \"81e5b54a-ff00-4c71-bedc-a8e07193cf69\" (UID: \"81e5b54a-ff00-4c71-bedc-a8e07193cf69\") "
Sep 18 18:48:56.615395 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.615370    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81e5b54a-ff00-4c71-bedc-a8e07193cf69-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "81e5b54a-ff00-4c71-bedc-a8e07193cf69" (UID: "81e5b54a-ff00-4c71-bedc-a8e07193cf69"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:48:56.615476 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.615397    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddab2017-6c34-41fd-b7b6-496228df51ce-kube-api-access-tzsrt" (OuterVolumeSpecName: "kube-api-access-tzsrt") pod "ddab2017-6c34-41fd-b7b6-496228df51ce" (UID: "ddab2017-6c34-41fd-b7b6-496228df51ce"). InnerVolumeSpecName "kube-api-access-tzsrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:48:56.615476 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.615459    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e5b54a-ff00-4c71-bedc-a8e07193cf69-kube-api-access-g26c6" (OuterVolumeSpecName: "kube-api-access-g26c6") pod "81e5b54a-ff00-4c71-bedc-a8e07193cf69" (UID: "81e5b54a-ff00-4c71-bedc-a8e07193cf69"). InnerVolumeSpecName "kube-api-access-g26c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:48:56.713531 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.713500    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzsrt\" (UniqueName: \"kubernetes.io/projected/ddab2017-6c34-41fd-b7b6-496228df51ce-kube-api-access-tzsrt\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:48:56.713531 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.713523    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g26c6\" (UniqueName: \"kubernetes.io/projected/81e5b54a-ff00-4c71-bedc-a8e07193cf69-kube-api-access-g26c6\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:48:56.713531 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.713534    2558 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81e5b54a-ff00-4c71-bedc-a8e07193cf69-tls-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:48:56.846909 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:56.846858    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fd94687d8-v9xrv"]
Sep 18 18:48:56.849085 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:48:56.849064    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83ed3401_aac9_48a9_9b6f_211e9db35d3e.slice/crio-34137a57045594fac1eacb3f646ee271a74f4ead592258b8e5f4ca087c18325d WatchSource:0}: Error finding container 34137a57045594fac1eacb3f646ee271a74f4ead592258b8e5f4ca087c18325d: Status 404 returned error can't find the container with id 34137a57045594fac1eacb3f646ee271a74f4ead592258b8e5f4ca087c18325d
Sep 18 18:48:57.494646 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:57.494544    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fd94687d8-v9xrv" event={"ID":"83ed3401-aac9-48a9-9b6f-211e9db35d3e","Type":"ContainerStarted","Data":"dbb2c7299c0c7f2c2a715a505c3e2c699e60b0e8d9d510fbdb06ec57bb18fd53"}
Sep 18 18:48:57.494646 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:57.494588    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fd94687d8-v9xrv" event={"ID":"83ed3401-aac9-48a9-9b6f-211e9db35d3e","Type":"ContainerStarted","Data":"34137a57045594fac1eacb3f646ee271a74f4ead592258b8e5f4ca087c18325d"}
Sep 18 18:48:57.494646 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:57.494588    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6w4b9"
Sep 18 18:48:57.495124 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:57.494728    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-8l829"
Sep 18 18:48:57.509592 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:57.509550    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5fd94687d8-v9xrv" podStartSLOduration=1.0318064200000001 podStartE2EDuration="1.509536032s" totalImagesPullingTime="477.729612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:48:57.507975423 +0000 UTC m=+596.944554303" watchObservedRunningTime="2026-09-18 18:48:57.509536032 +0000 UTC m=+596.946114874"
Sep 18 18:48:57.532966 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:57.532939    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6w4b9"]
Sep 18 18:48:57.542486 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:57.542454    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-6w4b9"]
Sep 18 18:48:57.565656 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:57.565627    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-8l829"]
Sep 18 18:48:57.570602 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:57.570576    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-8l829"]
Sep 18 18:48:58.575880 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:58.575836    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81e5b54a-ff00-4c71-bedc-a8e07193cf69" path="/var/lib/kubelet/pods/81e5b54a-ff00-4c71-bedc-a8e07193cf69/volumes"
Sep 18 18:48:58.576384 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:48:58.576165    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddab2017-6c34-41fd-b7b6-496228df51ce" path="/var/lib/kubelet/pods/ddab2017-6c34-41fd-b7b6-496228df51ce/volumes"
Sep 18 18:49:02.558679 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:02.558648    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 18:49:02.561156 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:02.561137    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 18:49:02.561586 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:02.561562    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 18:49:02.564148 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:02.564129    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 18:49:39.271162 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:39.271072    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-p97c7"]
Sep 18 18:49:39.275131 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:39.275107    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7"
Sep 18 18:49:39.277576 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:39.277549    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-qzgnc\""
Sep 18 18:49:39.277703 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:39.277597    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 18 18:49:39.284817 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:39.284766    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-p97c7"]
Sep 18 18:49:39.379898 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:39.379855    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb620bb-caa8-479f-832e-d7b6df868a82-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-p97c7\" (UID: \"2eb620bb-caa8-479f-832e-d7b6df868a82\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7"
Sep 18 18:49:39.380079 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:39.380016    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jx7g\" (UniqueName: \"kubernetes.io/projected/2eb620bb-caa8-479f-832e-d7b6df868a82-kube-api-access-4jx7g\") pod \"payload-processing-5b54b7fbb7-p97c7\" (UID: \"2eb620bb-caa8-479f-832e-d7b6df868a82\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7"
Sep 18 18:49:39.481399 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:39.481361    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb620bb-caa8-479f-832e-d7b6df868a82-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-p97c7\" (UID: \"2eb620bb-caa8-479f-832e-d7b6df868a82\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7"
Sep 18 18:49:39.481598 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:39.481463    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jx7g\" (UniqueName: \"kubernetes.io/projected/2eb620bb-caa8-479f-832e-d7b6df868a82-kube-api-access-4jx7g\") pod \"payload-processing-5b54b7fbb7-p97c7\" (UID: \"2eb620bb-caa8-479f-832e-d7b6df868a82\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7"
Sep 18 18:49:39.482402 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:39.482374    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb620bb-caa8-479f-832e-d7b6df868a82-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-p97c7\" (UID: \"2eb620bb-caa8-479f-832e-d7b6df868a82\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7"
Sep 18 18:49:39.493962 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:39.493936    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jx7g\" (UniqueName: \"kubernetes.io/projected/2eb620bb-caa8-479f-832e-d7b6df868a82-kube-api-access-4jx7g\") pod \"payload-processing-5b54b7fbb7-p97c7\" (UID: \"2eb620bb-caa8-479f-832e-d7b6df868a82\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7"
Sep 18 18:49:39.588802 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:39.588773    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7"
Sep 18 18:49:39.862923 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:39.862901    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-p97c7"]
Sep 18 18:49:39.864749 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:49:39.864709    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb620bb_caa8_479f_832e_d7b6df868a82.slice/crio-ba45b91ade72ea1c3629eb27c349885176d19af060f286970681f0e8142e1c54 WatchSource:0}: Error finding container ba45b91ade72ea1c3629eb27c349885176d19af060f286970681f0e8142e1c54: Status 404 returned error can't find the container with id ba45b91ade72ea1c3629eb27c349885176d19af060f286970681f0e8142e1c54
Sep 18 18:49:39.866761 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:39.866741    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:49:40.735740 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:40.735709    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7" event={"ID":"2eb620bb-caa8-479f-832e-d7b6df868a82","Type":"ContainerStarted","Data":"ba45b91ade72ea1c3629eb27c349885176d19af060f286970681f0e8142e1c54"}
Sep 18 18:49:41.751384 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:41.751345    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7" event={"ID":"2eb620bb-caa8-479f-832e-d7b6df868a82","Type":"ContainerStarted","Data":"c54be1b7af7f72af36c62cb8ea8637b0dc0ca67587f8d3741e34163a0ba8672b"}
Sep 18 18:49:41.751813 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:41.751502    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7"
Sep 18 18:49:41.768383 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:41.768341    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7" podStartSLOduration=1.43495752 podStartE2EDuration="2.768327055s" totalImagesPullingTime="1.333369535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:49:41.768031905 +0000 UTC m=+641.204610741" watchObservedRunningTime="2026-09-18 18:49:41.768327055 +0000 UTC m=+641.204905896"
Sep 18 18:49:52.759228 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:49:52.759200    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7"
Sep 18 18:50:11.447454 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.447419    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"]
Sep 18 18:50:11.451488 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.451465    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.454600 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.454575    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 18 18:50:11.454741 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.454605    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 18:50:11.454741 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.454576    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 18:50:11.454741 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.454576    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c6hs5\""
Sep 18 18:50:11.461406 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.461377    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"]
Sep 18 18:50:11.461898 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.461876    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.461981 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.461908    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.461981 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.461970    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.462087 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.461996    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbks7\" (UniqueName: \"kubernetes.io/projected/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-kube-api-access-mbks7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.462087 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.462037    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.462087 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.462053    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.563276 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.563229    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.563470 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.563286    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbks7\" (UniqueName: \"kubernetes.io/projected/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-kube-api-access-mbks7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.563470 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.563325    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.563470 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.563349    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.563470 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.563416    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.563470 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.563441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.563888 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.563860    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.563888 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.563878    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.564000 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.563944    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.565657 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.565635    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.566061 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.566040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.570773 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.570753    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbks7\" (UniqueName: \"kubernetes.io/projected/4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47-kube-api-access-mbks7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f5mmg\" (UID: \"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:11.762907 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:11.762815    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:12.041207 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:12.039452    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"]
Sep 18 18:50:12.927412 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:12.927369    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg" event={"ID":"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47","Type":"ContainerStarted","Data":"c839f9a38ab62ebb679b0c71b185dd3fcddb9aa523afa11c9a672c64c7ddd07b"}
Sep 18 18:50:13.934270 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:13.934234    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg" event={"ID":"4e1a3b5c-40fc-4f9b-96b6-eb8a45dfac47","Type":"ContainerStarted","Data":"86039d3fc8228666804898eea2010f8852ac4317ea9080d0eaccc7c27e102fbd"}
Sep 18 18:50:13.951021 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:13.950964    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg" podStartSLOduration=1.8036488880000001 podStartE2EDuration="2.950948643s" totalImagesPullingTime="1.147299755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:50:13.95051948 +0000 UTC m=+673.387098322" watchObservedRunningTime="2026-09-18 18:50:13.950948643 +0000 UTC m=+673.387527553"
Sep 18 18:50:21.763836 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:21.763792    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:21.764274 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:21.763874    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:21.776619 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:21.776598    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:21.988927 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:21.988898    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f5mmg"
Sep 18 18:50:36.472032 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.472000    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"]
Sep 18 18:50:36.475612 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.475596    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.478005 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.477986    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 18 18:50:36.485367 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.485343    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"]
Sep 18 18:50:36.560900 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.560869    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1113eee3-7583-4b23-88db-ac051c609725-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.561059 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.560912    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1113eee3-7583-4b23-88db-ac051c609725-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.561059 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.560945    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkp4z\" (UniqueName: \"kubernetes.io/projected/1113eee3-7583-4b23-88db-ac051c609725-kube-api-access-nkp4z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.561059 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.561021    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1113eee3-7583-4b23-88db-ac051c609725-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.561169 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.561069    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1113eee3-7583-4b23-88db-ac051c609725-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.561169 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.561095    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1113eee3-7583-4b23-88db-ac051c609725-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.661654 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.661615    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1113eee3-7583-4b23-88db-ac051c609725-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.661850 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.661692    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkp4z\" (UniqueName: \"kubernetes.io/projected/1113eee3-7583-4b23-88db-ac051c609725-kube-api-access-nkp4z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.661850 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.661718    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1113eee3-7583-4b23-88db-ac051c609725-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.661850 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.661790    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1113eee3-7583-4b23-88db-ac051c609725-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.661850 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.661812    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1113eee3-7583-4b23-88db-ac051c609725-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.662143 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.662122    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1113eee3-7583-4b23-88db-ac051c609725-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.662227 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.662137    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1113eee3-7583-4b23-88db-ac051c609725-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.662227 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.662168    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1113eee3-7583-4b23-88db-ac051c609725-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.662316 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.662235    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1113eee3-7583-4b23-88db-ac051c609725-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.664182 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.664160    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1113eee3-7583-4b23-88db-ac051c609725-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.664284 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.664250    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1113eee3-7583-4b23-88db-ac051c609725-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.669798 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.669774    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkp4z\" (UniqueName: \"kubernetes.io/projected/1113eee3-7583-4b23-88db-ac051c609725-kube-api-access-nkp4z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsx9h\" (UID: \"1113eee3-7583-4b23-88db-ac051c609725\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:36.787618 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:36.787538    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:37.059900 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:37.059877    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"]
Sep 18 18:50:37.061365 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:50:37.061342    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1113eee3_7583_4b23_88db_ac051c609725.slice/crio-24ab086fd15cc66d83d61e8ed38687e11477aaa9842161260a6fceabb835bf83 WatchSource:0}: Error finding container 24ab086fd15cc66d83d61e8ed38687e11477aaa9842161260a6fceabb835bf83: Status 404 returned error can't find the container with id 24ab086fd15cc66d83d61e8ed38687e11477aaa9842161260a6fceabb835bf83
Sep 18 18:50:38.067142 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:38.067101    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h" event={"ID":"1113eee3-7583-4b23-88db-ac051c609725","Type":"ContainerStarted","Data":"9e5c09fd5e6a2bf0f739bd2e93e033e7b18bf64162372440f8baab280b5ff765"}
Sep 18 18:50:38.067551 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:38.067178    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h" event={"ID":"1113eee3-7583-4b23-88db-ac051c609725","Type":"ContainerStarted","Data":"24ab086fd15cc66d83d61e8ed38687e11477aaa9842161260a6fceabb835bf83"}
Sep 18 18:50:38.085057 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:38.085002    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h" podStartSLOduration=1.912037416 podStartE2EDuration="2.084974101s" totalImagesPullingTime="172.936685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:50:38.083997566 +0000 UTC m=+697.520576410" watchObservedRunningTime="2026-09-18 18:50:38.084974101 +0000 UTC m=+697.521552943"
Sep 18 18:50:46.787912 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:46.787819    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:46.788326 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:46.787974    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:46.800651 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:46.800626    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:50:47.128328 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:50:47.128293    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsx9h"
Sep 18 18:51:19.449836 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.449796    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"]
Sep 18 18:51:19.454378 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.454354    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.456918 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.456898    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 18 18:51:19.464456 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.464431    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"]
Sep 18 18:51:19.539676 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.539634    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.539676 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.539675    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.539945 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.539793    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.539945 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.539849    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4q5\" (UniqueName: \"kubernetes.io/projected/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-kube-api-access-4g4q5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.539945 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.539885    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.539945 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.539933    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.640580 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.640542    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.640580 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.640583    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.640823 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.640604    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.640823 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.640730    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.641027 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.640970    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4q5\" (UniqueName: \"kubernetes.io/projected/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-kube-api-access-4g4q5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.641027 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.641015    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.641173 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.641022    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.641173 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.640980    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.641173 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.641067    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.642964 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.642939    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.643408 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.643388    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.649324 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.649297    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4q5\" (UniqueName: \"kubernetes.io/projected/a476d95c-59e5-40cf-b7ae-6e9b22afbd32-kube-api-access-4g4q5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t\" (UID: \"a476d95c-59e5-40cf-b7ae-6e9b22afbd32\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:19.764259 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:19.764154    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:20.038710 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:20.038494    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"]
Sep 18 18:51:20.301932 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:20.301905    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t" event={"ID":"a476d95c-59e5-40cf-b7ae-6e9b22afbd32","Type":"ContainerStarted","Data":"5a64b5ff6ec7609d1d8389163a6901d4bc33e1b8f8efeecc565c0b0c103e80d7"}
Sep 18 18:51:21.309071 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:21.309032    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t" event={"ID":"a476d95c-59e5-40cf-b7ae-6e9b22afbd32","Type":"ContainerStarted","Data":"292a011434f9993985138439e770af5ddbb90a79a66883049e4dfec0c3df6bdc"}
Sep 18 18:51:21.326340 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:21.326296    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t" podStartSLOduration=2.127226364 podStartE2EDuration="2.326281037s" totalImagesPullingTime="199.054673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:51:21.324980426 +0000 UTC m=+740.761559269" watchObservedRunningTime="2026-09-18 18:51:21.326281037 +0000 UTC m=+740.762859879"
Sep 18 18:51:29.764302 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:29.764268    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:29.764766 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:29.764350    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:29.776995 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:29.776968    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:51:30.382164 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:51:30.382136    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7fj8t"
Sep 18 18:52:23.603702 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:23.603624    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fd94687d8-v9xrv"]
Sep 18 18:52:23.604205 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:23.603912    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5fd94687d8-v9xrv" podUID="83ed3401-aac9-48a9-9b6f-211e9db35d3e" containerName="authorino" containerID="cri-o://dbb2c7299c0c7f2c2a715a505c3e2c699e60b0e8d9d510fbdb06ec57bb18fd53" gracePeriod=30
Sep 18 18:52:23.857388 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:23.857324    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fd94687d8-v9xrv"
Sep 18 18:52:23.909275 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:23.909248    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83ed3401-aac9-48a9-9b6f-211e9db35d3e-tls-cert\") pod \"83ed3401-aac9-48a9-9b6f-211e9db35d3e\" (UID: \"83ed3401-aac9-48a9-9b6f-211e9db35d3e\") "
Sep 18 18:52:23.909447 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:23.909290    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v89cr\" (UniqueName: \"kubernetes.io/projected/83ed3401-aac9-48a9-9b6f-211e9db35d3e-kube-api-access-v89cr\") pod \"83ed3401-aac9-48a9-9b6f-211e9db35d3e\" (UID: \"83ed3401-aac9-48a9-9b6f-211e9db35d3e\") "
Sep 18 18:52:23.911384 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:23.911359    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ed3401-aac9-48a9-9b6f-211e9db35d3e-kube-api-access-v89cr" (OuterVolumeSpecName: "kube-api-access-v89cr") pod "83ed3401-aac9-48a9-9b6f-211e9db35d3e" (UID: "83ed3401-aac9-48a9-9b6f-211e9db35d3e"). InnerVolumeSpecName "kube-api-access-v89cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:52:23.919444 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:23.919417    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ed3401-aac9-48a9-9b6f-211e9db35d3e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "83ed3401-aac9-48a9-9b6f-211e9db35d3e" (UID: "83ed3401-aac9-48a9-9b6f-211e9db35d3e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:52:24.010341 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.010308    2558 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83ed3401-aac9-48a9-9b6f-211e9db35d3e-tls-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:52:24.010341 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.010337    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v89cr\" (UniqueName: \"kubernetes.io/projected/83ed3401-aac9-48a9-9b6f-211e9db35d3e-kube-api-access-v89cr\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:52:24.337524 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.337496    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-p97c7"]
Sep 18 18:52:24.337789 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.337737    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7" podUID="2eb620bb-caa8-479f-832e-d7b6df868a82" containerName="payload-processing" containerID="cri-o://c54be1b7af7f72af36c62cb8ea8637b0dc0ca67587f8d3741e34163a0ba8672b" gracePeriod=30
Sep 18 18:52:24.572902 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.572877    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7"
Sep 18 18:52:24.615233 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.615141    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb620bb-caa8-479f-832e-d7b6df868a82-plugins-config-volume\") pod \"2eb620bb-caa8-479f-832e-d7b6df868a82\" (UID: \"2eb620bb-caa8-479f-832e-d7b6df868a82\") "
Sep 18 18:52:24.615233 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.615218    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jx7g\" (UniqueName: \"kubernetes.io/projected/2eb620bb-caa8-479f-832e-d7b6df868a82-kube-api-access-4jx7g\") pod \"2eb620bb-caa8-479f-832e-d7b6df868a82\" (UID: \"2eb620bb-caa8-479f-832e-d7b6df868a82\") "
Sep 18 18:52:24.615635 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.615592    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb620bb-caa8-479f-832e-d7b6df868a82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2eb620bb-caa8-479f-832e-d7b6df868a82" (UID: "2eb620bb-caa8-479f-832e-d7b6df868a82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:52:24.617196 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.617162    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb620bb-caa8-479f-832e-d7b6df868a82-kube-api-access-4jx7g" (OuterVolumeSpecName: "kube-api-access-4jx7g") pod "2eb620bb-caa8-479f-832e-d7b6df868a82" (UID: "2eb620bb-caa8-479f-832e-d7b6df868a82"). InnerVolumeSpecName "kube-api-access-4jx7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:52:24.677122 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.677087    2558 generic.go:356] "Generic (PLEG): container finished" podID="83ed3401-aac9-48a9-9b6f-211e9db35d3e" containerID="dbb2c7299c0c7f2c2a715a505c3e2c699e60b0e8d9d510fbdb06ec57bb18fd53" exitCode=0
Sep 18 18:52:24.677322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.677138    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fd94687d8-v9xrv"
Sep 18 18:52:24.677322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.677176    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fd94687d8-v9xrv" event={"ID":"83ed3401-aac9-48a9-9b6f-211e9db35d3e","Type":"ContainerDied","Data":"dbb2c7299c0c7f2c2a715a505c3e2c699e60b0e8d9d510fbdb06ec57bb18fd53"}
Sep 18 18:52:24.677322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.677253    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fd94687d8-v9xrv" event={"ID":"83ed3401-aac9-48a9-9b6f-211e9db35d3e","Type":"ContainerDied","Data":"34137a57045594fac1eacb3f646ee271a74f4ead592258b8e5f4ca087c18325d"}
Sep 18 18:52:24.677322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.677274    2558 scope.go:117] "RemoveContainer" containerID="dbb2c7299c0c7f2c2a715a505c3e2c699e60b0e8d9d510fbdb06ec57bb18fd53"
Sep 18 18:52:24.678901 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.678823    2558 generic.go:356] "Generic (PLEG): container finished" podID="2eb620bb-caa8-479f-832e-d7b6df868a82" containerID="c54be1b7af7f72af36c62cb8ea8637b0dc0ca67587f8d3741e34163a0ba8672b" exitCode=0
Sep 18 18:52:24.678901 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.678882    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7"
Sep 18 18:52:24.678901 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.678885    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7" event={"ID":"2eb620bb-caa8-479f-832e-d7b6df868a82","Type":"ContainerDied","Data":"c54be1b7af7f72af36c62cb8ea8637b0dc0ca67587f8d3741e34163a0ba8672b"}
Sep 18 18:52:24.679099 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.678924    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-p97c7" event={"ID":"2eb620bb-caa8-479f-832e-d7b6df868a82","Type":"ContainerDied","Data":"ba45b91ade72ea1c3629eb27c349885176d19af060f286970681f0e8142e1c54"}
Sep 18 18:52:24.689011 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.688939    2558 scope.go:117] "RemoveContainer" containerID="dbb2c7299c0c7f2c2a715a505c3e2c699e60b0e8d9d510fbdb06ec57bb18fd53"
Sep 18 18:52:24.690506 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:52:24.690367    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb2c7299c0c7f2c2a715a505c3e2c699e60b0e8d9d510fbdb06ec57bb18fd53\": container with ID starting with dbb2c7299c0c7f2c2a715a505c3e2c699e60b0e8d9d510fbdb06ec57bb18fd53 not found: ID does not exist" containerID="dbb2c7299c0c7f2c2a715a505c3e2c699e60b0e8d9d510fbdb06ec57bb18fd53"
Sep 18 18:52:24.690506 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.690405    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb2c7299c0c7f2c2a715a505c3e2c699e60b0e8d9d510fbdb06ec57bb18fd53"} err="failed to get container status \"dbb2c7299c0c7f2c2a715a505c3e2c699e60b0e8d9d510fbdb06ec57bb18fd53\": rpc error: code = NotFound desc = could not find container \"dbb2c7299c0c7f2c2a715a505c3e2c699e60b0e8d9d510fbdb06ec57bb18fd53\": container with ID starting with dbb2c7299c0c7f2c2a715a505c3e2c699e60b0e8d9d510fbdb06ec57bb18fd53 not found: ID does not exist"
Sep 18 18:52:24.690506 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.690425    2558 scope.go:117] "RemoveContainer" containerID="c54be1b7af7f72af36c62cb8ea8637b0dc0ca67587f8d3741e34163a0ba8672b"
Sep 18 18:52:24.699596 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.699570    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fd94687d8-v9xrv"]
Sep 18 18:52:24.700211 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.700174    2558 scope.go:117] "RemoveContainer" containerID="c54be1b7af7f72af36c62cb8ea8637b0dc0ca67587f8d3741e34163a0ba8672b"
Sep 18 18:52:24.700763 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:52:24.700694    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c54be1b7af7f72af36c62cb8ea8637b0dc0ca67587f8d3741e34163a0ba8672b\": container with ID starting with c54be1b7af7f72af36c62cb8ea8637b0dc0ca67587f8d3741e34163a0ba8672b not found: ID does not exist" containerID="c54be1b7af7f72af36c62cb8ea8637b0dc0ca67587f8d3741e34163a0ba8672b"
Sep 18 18:52:24.700861 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.700765    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54be1b7af7f72af36c62cb8ea8637b0dc0ca67587f8d3741e34163a0ba8672b"} err="failed to get container status \"c54be1b7af7f72af36c62cb8ea8637b0dc0ca67587f8d3741e34163a0ba8672b\": rpc error: code = NotFound desc = could not find container \"c54be1b7af7f72af36c62cb8ea8637b0dc0ca67587f8d3741e34163a0ba8672b\": container with ID starting with c54be1b7af7f72af36c62cb8ea8637b0dc0ca67587f8d3741e34163a0ba8672b not found: ID does not exist"
Sep 18 18:52:24.702975 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.702954    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5fd94687d8-v9xrv"]
Sep 18 18:52:24.712940 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.712918    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-p97c7"]
Sep 18 18:52:24.716028 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.715990    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb620bb-caa8-479f-832e-d7b6df868a82-plugins-config-volume\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:52:24.716028 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.716018    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jx7g\" (UniqueName: \"kubernetes.io/projected/2eb620bb-caa8-479f-832e-d7b6df868a82-kube-api-access-4jx7g\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:52:24.718600 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:24.718580    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-p97c7"]
Sep 18 18:52:26.575786 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:26.575743    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eb620bb-caa8-479f-832e-d7b6df868a82" path="/var/lib/kubelet/pods/2eb620bb-caa8-479f-832e-d7b6df868a82/volumes"
Sep 18 18:52:26.576304 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:26.576263    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83ed3401-aac9-48a9-9b6f-211e9db35d3e" path="/var/lib/kubelet/pods/83ed3401-aac9-48a9-9b6f-211e9db35d3e/volumes"
Sep 18 18:52:37.858402 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:37.858366    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-58d5755467-xg7kg"]
Sep 18 18:52:37.858855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:37.858748    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ed3401-aac9-48a9-9b6f-211e9db35d3e" containerName="authorino"
Sep 18 18:52:37.858855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:37.858759    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ed3401-aac9-48a9-9b6f-211e9db35d3e" containerName="authorino"
Sep 18 18:52:37.858855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:37.858767    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb620bb-caa8-479f-832e-d7b6df868a82" containerName="payload-processing"
Sep 18 18:52:37.858855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:37.858773    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb620bb-caa8-479f-832e-d7b6df868a82" containerName="payload-processing"
Sep 18 18:52:37.858855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:37.858843    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ed3401-aac9-48a9-9b6f-211e9db35d3e" containerName="authorino"
Sep 18 18:52:37.858855 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:37.858849    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb620bb-caa8-479f-832e-d7b6df868a82" containerName="payload-processing"
Sep 18 18:52:37.863577 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:37.863561    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-58d5755467-xg7kg"
Sep 18 18:52:37.867240 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:37.867170    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Sep 18 18:52:37.867366 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:37.867211    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-mdfrk\""
Sep 18 18:52:37.879339 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:37.879315    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-58d5755467-xg7kg"]
Sep 18 18:52:37.934439 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:37.934409    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m5f7\" (UniqueName: \"kubernetes.io/projected/b3f684c3-aa60-4ad2-990e-c644be7356df-kube-api-access-8m5f7\") pod \"ai-gateway-controller-58d5755467-xg7kg\" (UID: \"b3f684c3-aa60-4ad2-990e-c644be7356df\") " pod="opendatahub/ai-gateway-controller-58d5755467-xg7kg"
Sep 18 18:52:38.035328 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:38.035291    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m5f7\" (UniqueName: \"kubernetes.io/projected/b3f684c3-aa60-4ad2-990e-c644be7356df-kube-api-access-8m5f7\") pod \"ai-gateway-controller-58d5755467-xg7kg\" (UID: \"b3f684c3-aa60-4ad2-990e-c644be7356df\") " pod="opendatahub/ai-gateway-controller-58d5755467-xg7kg"
Sep 18 18:52:38.042995 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:38.042971    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m5f7\" (UniqueName: \"kubernetes.io/projected/b3f684c3-aa60-4ad2-990e-c644be7356df-kube-api-access-8m5f7\") pod \"ai-gateway-controller-58d5755467-xg7kg\" (UID: \"b3f684c3-aa60-4ad2-990e-c644be7356df\") " pod="opendatahub/ai-gateway-controller-58d5755467-xg7kg"
Sep 18 18:52:38.173899 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:38.173810    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-58d5755467-xg7kg"
Sep 18 18:52:38.445698 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:38.445672    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-58d5755467-xg7kg"]
Sep 18 18:52:38.447336 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:52:38.447309    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f684c3_aa60_4ad2_990e_c644be7356df.slice/crio-759ffd509cd31e17abdfb12386e5fde933acddf2c85b32aa4bf7f9f818e03055 WatchSource:0}: Error finding container 759ffd509cd31e17abdfb12386e5fde933acddf2c85b32aa4bf7f9f818e03055: Status 404 returned error can't find the container with id 759ffd509cd31e17abdfb12386e5fde933acddf2c85b32aa4bf7f9f818e03055
Sep 18 18:52:38.758955 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:38.758877    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-58d5755467-xg7kg" event={"ID":"b3f684c3-aa60-4ad2-990e-c644be7356df","Type":"ContainerStarted","Data":"759ffd509cd31e17abdfb12386e5fde933acddf2c85b32aa4bf7f9f818e03055"}
Sep 18 18:52:41.781695 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:41.781660    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-58d5755467-xg7kg" event={"ID":"b3f684c3-aa60-4ad2-990e-c644be7356df","Type":"ContainerStarted","Data":"7b6cac3ba2777b20bd05db53e9827a4752290bb2e5a159238250cdb2e95a0a03"}
Sep 18 18:52:41.782114 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:41.781813    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-58d5755467-xg7kg"
Sep 18 18:52:41.798333 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:41.798282    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-58d5755467-xg7kg" podStartSLOduration=2.520116966 podStartE2EDuration="4.79826478s" totalImagesPullingTime="2.278147814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:52:41.796452751 +0000 UTC m=+821.233031595" watchObservedRunningTime="2026-09-18 18:52:41.79826478 +0000 UTC m=+821.234843621"
Sep 18 18:52:46.516570 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.516532    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-85fd65c94c-bbxqv"]
Sep 18 18:52:46.520160 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.520141    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-85fd65c94c-bbxqv"
Sep 18 18:52:46.522740 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.522720    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pvgdd\""
Sep 18 18:52:46.524007 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.523991    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 18 18:52:46.532694 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.532670    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-85fd65c94c-bbxqv"]
Sep 18 18:52:46.535724 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.535705    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs"]
Sep 18 18:52:46.539146 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.539131    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs"
Sep 18 18:52:46.546896 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.546876    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs"]
Sep 18 18:52:46.626572 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.626541    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b22143-f65e-4cf9-8674-14fe8946d89f-plugins-config-volume\") pod \"payload-pre-processing-76f657cdc4-fvbcs\" (UID: \"a0b22143-f65e-4cf9-8674-14fe8946d89f\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs"
Sep 18 18:52:46.626755 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.626595    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdqg6\" (UniqueName: \"kubernetes.io/projected/ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1-kube-api-access-rdqg6\") pod \"payload-processing-85fd65c94c-bbxqv\" (UID: \"ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1\") " pod="openshift-ingress/payload-processing-85fd65c94c-bbxqv"
Sep 18 18:52:46.626755 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.626620    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbsdz\" (UniqueName: \"kubernetes.io/projected/a0b22143-f65e-4cf9-8674-14fe8946d89f-kube-api-access-sbsdz\") pod \"payload-pre-processing-76f657cdc4-fvbcs\" (UID: \"a0b22143-f65e-4cf9-8674-14fe8946d89f\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs"
Sep 18 18:52:46.626755 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.626685    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1-plugins-config-volume\") pod \"payload-processing-85fd65c94c-bbxqv\" (UID: \"ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1\") " pod="openshift-ingress/payload-processing-85fd65c94c-bbxqv"
Sep 18 18:52:46.728078 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.728037    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1-plugins-config-volume\") pod \"payload-processing-85fd65c94c-bbxqv\" (UID: \"ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1\") " pod="openshift-ingress/payload-processing-85fd65c94c-bbxqv"
Sep 18 18:52:46.728312 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.728137    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b22143-f65e-4cf9-8674-14fe8946d89f-plugins-config-volume\") pod \"payload-pre-processing-76f657cdc4-fvbcs\" (UID: \"a0b22143-f65e-4cf9-8674-14fe8946d89f\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs"
Sep 18 18:52:46.728312 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.728221    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdqg6\" (UniqueName: \"kubernetes.io/projected/ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1-kube-api-access-rdqg6\") pod \"payload-processing-85fd65c94c-bbxqv\" (UID: \"ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1\") " pod="openshift-ingress/payload-processing-85fd65c94c-bbxqv"
Sep 18 18:52:46.728312 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.728246    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbsdz\" (UniqueName: \"kubernetes.io/projected/a0b22143-f65e-4cf9-8674-14fe8946d89f-kube-api-access-sbsdz\") pod \"payload-pre-processing-76f657cdc4-fvbcs\" (UID: \"a0b22143-f65e-4cf9-8674-14fe8946d89f\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs"
Sep 18 18:52:46.729055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.729029    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b22143-f65e-4cf9-8674-14fe8946d89f-plugins-config-volume\") pod \"payload-pre-processing-76f657cdc4-fvbcs\" (UID: \"a0b22143-f65e-4cf9-8674-14fe8946d89f\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs"
Sep 18 18:52:46.729158 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.729041    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1-plugins-config-volume\") pod \"payload-processing-85fd65c94c-bbxqv\" (UID: \"ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1\") " pod="openshift-ingress/payload-processing-85fd65c94c-bbxqv"
Sep 18 18:52:46.736076 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.736053    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdqg6\" (UniqueName: \"kubernetes.io/projected/ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1-kube-api-access-rdqg6\") pod \"payload-processing-85fd65c94c-bbxqv\" (UID: \"ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1\") " pod="openshift-ingress/payload-processing-85fd65c94c-bbxqv"
Sep 18 18:52:46.736277 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.736255    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbsdz\" (UniqueName: \"kubernetes.io/projected/a0b22143-f65e-4cf9-8674-14fe8946d89f-kube-api-access-sbsdz\") pod \"payload-pre-processing-76f657cdc4-fvbcs\" (UID: \"a0b22143-f65e-4cf9-8674-14fe8946d89f\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs"
Sep 18 18:52:46.831222 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.831106    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-85fd65c94c-bbxqv"
Sep 18 18:52:46.850756 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:46.850728    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs"
Sep 18 18:52:47.104395 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:47.104366    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-85fd65c94c-bbxqv"]
Sep 18 18:52:47.106911 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:52:47.106886    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba97ff99_c0a3_474d_9697_0fb7c0f8dbb1.slice/crio-6899a9c5de73cffb170ecf1fae76a562ca7ec0394785820fde8d5e057de926ca WatchSource:0}: Error finding container 6899a9c5de73cffb170ecf1fae76a562ca7ec0394785820fde8d5e057de926ca: Status 404 returned error can't find the container with id 6899a9c5de73cffb170ecf1fae76a562ca7ec0394785820fde8d5e057de926ca
Sep 18 18:52:47.138406 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:47.138383    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs"]
Sep 18 18:52:47.139659 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:52:47.139630    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b22143_f65e_4cf9_8674_14fe8946d89f.slice/crio-601ecf82647b2e25c8732bee1ffb0dd9dca7deabe084c0a3fec549657c3bba38 WatchSource:0}: Error finding container 601ecf82647b2e25c8732bee1ffb0dd9dca7deabe084c0a3fec549657c3bba38: Status 404 returned error can't find the container with id 601ecf82647b2e25c8732bee1ffb0dd9dca7deabe084c0a3fec549657c3bba38
Sep 18 18:52:47.821943 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:47.821908    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-85fd65c94c-bbxqv" event={"ID":"ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1","Type":"ContainerStarted","Data":"6899a9c5de73cffb170ecf1fae76a562ca7ec0394785820fde8d5e057de926ca"}
Sep 18 18:52:47.823649 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:47.823623    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs" event={"ID":"a0b22143-f65e-4cf9-8674-14fe8946d89f","Type":"ContainerStarted","Data":"601ecf82647b2e25c8732bee1ffb0dd9dca7deabe084c0a3fec549657c3bba38"}
Sep 18 18:52:48.833458 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:48.833357    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs" event={"ID":"a0b22143-f65e-4cf9-8674-14fe8946d89f","Type":"ContainerStarted","Data":"104a66229f008f995c11be6ca32da178c45d4e7227944fb2aa43c28eff2b9384"}
Sep 18 18:52:48.833906 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:48.833529    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs"
Sep 18 18:52:48.834916 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:48.834897    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-85fd65c94c-bbxqv" event={"ID":"ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1","Type":"ContainerStarted","Data":"256f3ee838dc1d0637ae5079489ad41864c9f06f87213a3dfcf17e0a2cfd1667"}
Sep 18 18:52:48.834996 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:48.834935    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-85fd65c94c-bbxqv"
Sep 18 18:52:48.850976 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:48.850936    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs" podStartSLOduration=1.61353762 podStartE2EDuration="2.850920709s" totalImagesPullingTime="1.237383089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:52:48.849521457 +0000 UTC m=+828.286100300" watchObservedRunningTime="2026-09-18 18:52:48.850920709 +0000 UTC m=+828.287499548"
Sep 18 18:52:48.866127 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:48.866081    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-85fd65c94c-bbxqv" podStartSLOduration=1.597922056 podStartE2EDuration="2.866066668s" totalImagesPullingTime="1.268144612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:52:48.863837957 +0000 UTC m=+828.300416800" watchObservedRunningTime="2026-09-18 18:52:48.866066668 +0000 UTC m=+828.302645510"
Sep 18 18:52:52.791558 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:52.791524    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-58d5755467-xg7kg"
Sep 18 18:52:59.842599 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:59.842565    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-76f657cdc4-fvbcs"
Sep 18 18:52:59.843023 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:52:59.842649    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-85fd65c94c-bbxqv"
Sep 18 18:53:06.890073 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:06.890036    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-58d5755467-xg7kg"]
Sep 18 18:53:06.890570 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:06.890264    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-58d5755467-xg7kg" podUID="b3f684c3-aa60-4ad2-990e-c644be7356df" containerName="manager" containerID="cri-o://7b6cac3ba2777b20bd05db53e9827a4752290bb2e5a159238250cdb2e95a0a03" gracePeriod=30
Sep 18 18:53:07.136837 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:07.136814    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-58d5755467-xg7kg"
Sep 18 18:53:07.203530 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:07.203446    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m5f7\" (UniqueName: \"kubernetes.io/projected/b3f684c3-aa60-4ad2-990e-c644be7356df-kube-api-access-8m5f7\") pod \"b3f684c3-aa60-4ad2-990e-c644be7356df\" (UID: \"b3f684c3-aa60-4ad2-990e-c644be7356df\") "
Sep 18 18:53:07.205610 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:07.205583    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f684c3-aa60-4ad2-990e-c644be7356df-kube-api-access-8m5f7" (OuterVolumeSpecName: "kube-api-access-8m5f7") pod "b3f684c3-aa60-4ad2-990e-c644be7356df" (UID: "b3f684c3-aa60-4ad2-990e-c644be7356df"). InnerVolumeSpecName "kube-api-access-8m5f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:53:07.304488 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:07.304453    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m5f7\" (UniqueName: \"kubernetes.io/projected/b3f684c3-aa60-4ad2-990e-c644be7356df-kube-api-access-8m5f7\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:53:07.943940 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:07.943907    2558 generic.go:356] "Generic (PLEG): container finished" podID="b3f684c3-aa60-4ad2-990e-c644be7356df" containerID="7b6cac3ba2777b20bd05db53e9827a4752290bb2e5a159238250cdb2e95a0a03" exitCode=0
Sep 18 18:53:07.944436 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:07.943962    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-58d5755467-xg7kg"
Sep 18 18:53:07.944436 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:07.944007    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-58d5755467-xg7kg" event={"ID":"b3f684c3-aa60-4ad2-990e-c644be7356df","Type":"ContainerDied","Data":"7b6cac3ba2777b20bd05db53e9827a4752290bb2e5a159238250cdb2e95a0a03"}
Sep 18 18:53:07.944436 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:07.944060    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-58d5755467-xg7kg" event={"ID":"b3f684c3-aa60-4ad2-990e-c644be7356df","Type":"ContainerDied","Data":"759ffd509cd31e17abdfb12386e5fde933acddf2c85b32aa4bf7f9f818e03055"}
Sep 18 18:53:07.944436 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:07.944083    2558 scope.go:117] "RemoveContainer" containerID="7b6cac3ba2777b20bd05db53e9827a4752290bb2e5a159238250cdb2e95a0a03"
Sep 18 18:53:07.953369 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:07.953350    2558 scope.go:117] "RemoveContainer" containerID="7b6cac3ba2777b20bd05db53e9827a4752290bb2e5a159238250cdb2e95a0a03"
Sep 18 18:53:07.953640 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:53:07.953621    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6cac3ba2777b20bd05db53e9827a4752290bb2e5a159238250cdb2e95a0a03\": container with ID starting with 7b6cac3ba2777b20bd05db53e9827a4752290bb2e5a159238250cdb2e95a0a03 not found: ID does not exist" containerID="7b6cac3ba2777b20bd05db53e9827a4752290bb2e5a159238250cdb2e95a0a03"
Sep 18 18:53:07.953691 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:07.953647    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6cac3ba2777b20bd05db53e9827a4752290bb2e5a159238250cdb2e95a0a03"} err="failed to get container status \"7b6cac3ba2777b20bd05db53e9827a4752290bb2e5a159238250cdb2e95a0a03\": rpc error: code = NotFound desc = could not find container \"7b6cac3ba2777b20bd05db53e9827a4752290bb2e5a159238250cdb2e95a0a03\": container with ID starting with 7b6cac3ba2777b20bd05db53e9827a4752290bb2e5a159238250cdb2e95a0a03 not found: ID does not exist"
Sep 18 18:53:07.965473 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:07.965451    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-58d5755467-xg7kg"]
Sep 18 18:53:07.969199 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:07.969169    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-58d5755467-xg7kg"]
Sep 18 18:53:08.575984 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:08.575955    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3f684c3-aa60-4ad2-990e-c644be7356df" path="/var/lib/kubelet/pods/b3f684c3-aa60-4ad2-990e-c644be7356df/volumes"
Sep 18 18:53:24.994221 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:24.994169    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"]
Sep 18 18:53:24.995166 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:24.994638    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f684c3-aa60-4ad2-990e-c644be7356df" containerName="manager"
Sep 18 18:53:24.995166 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:24.994651    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f684c3-aa60-4ad2-990e-c644be7356df" containerName="manager"
Sep 18 18:53:24.995166 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:24.994730    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3f684c3-aa60-4ad2-990e-c644be7356df" containerName="manager"
Sep 18 18:53:24.999575 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:24.999554    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.002000 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.001984    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9d9f36-openshift-default-dockercfg-j67g8\""
Sep 18 18:53:25.018153 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.018128    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"]
Sep 18 18:53:25.061166 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.061124    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istiod-ca-cert\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.061377 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.061228    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-credential-socket\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.061377 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.061262    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gcwb\" (UniqueName: \"kubernetes.io/projected/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-kube-api-access-4gcwb\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.061377 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.061294    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-workload-socket\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.061377 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.061360    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-token\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.061602 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.061432    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-workload-certs\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.061602 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.061490    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-data\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.061602 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.061525    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-podinfo\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.061754 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.061650    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-envoy\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.163205 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.162830    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-envoy\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.163205 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.162885    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istiod-ca-cert\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.163205 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.162940    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-credential-socket\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.163205 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.162967    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcwb\" (UniqueName: \"kubernetes.io/projected/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-kube-api-access-4gcwb\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.163205 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.163000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-workload-socket\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.163205 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.163038    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-token\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.163205 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.163112    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-workload-certs\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.163205 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.163139    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-data\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.163205 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.163159    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-podinfo\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.165818 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.165785    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-podinfo\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.166776 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.166747    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-workload-socket\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.166887 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.166773    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-workload-certs\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.167024 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.167001    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-credential-socket\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.168014 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.167763    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istiod-ca-cert\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.168014 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.167832    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-data\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.169520 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.169492    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-envoy\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.178372 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.178335    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"]
Sep 18 18:53:25.183937 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.183916    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gcwb\" (UniqueName: \"kubernetes.io/projected/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-kube-api-access-4gcwb\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.184020 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.183969    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-token\") pod \"e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") " pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.185353 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.185335    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.188632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.188609    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-cc289c-openshift-default-dockercfg-tkwjz\""
Sep 18 18:53:25.195967 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.195947    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"]
Sep 18 18:53:25.264048 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.263964    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-envoy\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.264048 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.263999    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-token\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.264048 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.264025    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-credential-socket\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.264346 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.264169    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-podinfo\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.264346 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.264244    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-workload-socket\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.264346 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.264271    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-workload-certs\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.264346 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.264298    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-data\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.264346 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.264325    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9kqx\" (UniqueName: \"kubernetes.io/projected/42bf6cd3-467b-4c9f-9960-c1214c476a44-kube-api-access-l9kqx\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.264527 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.264372    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42bf6cd3-467b-4c9f-9960-c1214c476a44-istiod-ca-cert\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.311941 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.311907    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:25.364911 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.364877    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-envoy\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.364911 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.364912    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-token\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.365105 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.364934    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-credential-socket\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.365105 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.364979    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-podinfo\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.365105 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.365006    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-workload-socket\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.365105 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.365020    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-workload-certs\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.365105 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.365039    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-data\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.365105 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.365056    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9kqx\" (UniqueName: \"kubernetes.io/projected/42bf6cd3-467b-4c9f-9960-c1214c476a44-kube-api-access-l9kqx\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.365105 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.365084    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42bf6cd3-467b-4c9f-9960-c1214c476a44-istiod-ca-cert\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.365624 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.365591    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-workload-socket\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.365914 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.365888    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42bf6cd3-467b-4c9f-9960-c1214c476a44-istiod-ca-cert\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.366254 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.366156    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-workload-certs\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.366551 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.366462    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-data\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.366551 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.366491    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-credential-socket\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.367989 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.367612    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-envoy\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.368566 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.368510    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-podinfo\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.376611 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.376561    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-token\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.377055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.377003    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9kqx\" (UniqueName: \"kubernetes.io/projected/42bf6cd3-467b-4c9f-9960-c1214c476a44-kube-api-access-l9kqx\") pod \"e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") " pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.515960 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.515868    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:25.600862 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.600830    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"]
Sep 18 18:53:25.605345 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:53:25.605314    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a468882_6c2c_4d9a_87c5_8e4b3e5868ab.slice/crio-cb9dde1182837d28bea0f17538dd0d0eeb28695cdf93661d1d63631ff4de5af1 WatchSource:0}: Error finding container cb9dde1182837d28bea0f17538dd0d0eeb28695cdf93661d1d63631ff4de5af1: Status 404 returned error can't find the container with id cb9dde1182837d28bea0f17538dd0d0eeb28695cdf93661d1d63631ff4de5af1
Sep 18 18:53:25.607223 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.607167    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:25.607320 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.607246    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:25.607320 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.607278    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:25.793928 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.793902    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"]
Sep 18 18:53:25.795402 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:53:25.795372    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42bf6cd3_467b_4c9f_9960_c1214c476a44.slice/crio-74ff2804920fa7df62016ec89ad10c7b0f0c4d3c2b2e7cfb43bb65e9195a778a WatchSource:0}: Error finding container 74ff2804920fa7df62016ec89ad10c7b0f0c4d3c2b2e7cfb43bb65e9195a778a: Status 404 returned error can't find the container with id 74ff2804920fa7df62016ec89ad10c7b0f0c4d3c2b2e7cfb43bb65e9195a778a
Sep 18 18:53:25.797689 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.797657    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:25.797766 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.797719    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:25.797766 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:25.797748    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:26.047127 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:26.047003    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z" event={"ID":"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab","Type":"ContainerStarted","Data":"b04bde102f863f93aaf8816675e756f2ab29323de7f709896b5e8e3025f1baac"}
Sep 18 18:53:26.047127 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:26.047086    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z" event={"ID":"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab","Type":"ContainerStarted","Data":"cb9dde1182837d28bea0f17538dd0d0eeb28695cdf93661d1d63631ff4de5af1"}
Sep 18 18:53:26.049217 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:26.049154    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj" event={"ID":"42bf6cd3-467b-4c9f-9960-c1214c476a44","Type":"ContainerStarted","Data":"2844f9c5b2702140a5cd8766edf6a93dbe0836d9351485f62b573bd779489c5d"}
Sep 18 18:53:26.049359 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:26.049223    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj" event={"ID":"42bf6cd3-467b-4c9f-9960-c1214c476a44","Type":"ContainerStarted","Data":"74ff2804920fa7df62016ec89ad10c7b0f0c4d3c2b2e7cfb43bb65e9195a778a"}
Sep 18 18:53:26.071944 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:26.071888    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z" podStartSLOduration=2.071871794 podStartE2EDuration="2.071871794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:53:26.070442712 +0000 UTC m=+865.507021579" watchObservedRunningTime="2026-09-18 18:53:26.071871794 +0000 UTC m=+865.508450635"
Sep 18 18:53:26.093886 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:26.093831    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj" podStartSLOduration=1.093816766 podStartE2EDuration="1.093816766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:53:26.090853794 +0000 UTC m=+865.527432648" watchObservedRunningTime="2026-09-18 18:53:26.093816766 +0000 UTC m=+865.530395607"
Sep 18 18:53:26.312808 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:26.312718    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:26.318130 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:26.318106    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:26.516108 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:26.516068    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:26.521373 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:26.521348    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:27.055202 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:27.055150    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:27.055659 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:27.055222    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:27.056144 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:27.056119    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:27.056267 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:27.056177    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:30.607928 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:30.607896    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8962-ce8d10"]
Sep 18 18:53:30.612051 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:30.612034    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8962-ce8d10"
Sep 18 18:53:30.614438 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:30.614418    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7m4wl\""
Sep 18 18:53:30.618482 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:30.618462    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8962-ce8d10"]
Sep 18 18:53:30.713858 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:30.713826    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-752kx\" (UniqueName: \"kubernetes.io/projected/972e51c6-627c-4fe1-9d3c-aa7c89f0b26b-kube-api-access-752kx\") pod \"test-curl-8962-ce8d10\" (UID: \"972e51c6-627c-4fe1-9d3c-aa7c89f0b26b\") " pod="opendatahub/test-curl-8962-ce8d10"
Sep 18 18:53:30.815012 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:30.814971    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-752kx\" (UniqueName: \"kubernetes.io/projected/972e51c6-627c-4fe1-9d3c-aa7c89f0b26b-kube-api-access-752kx\") pod \"test-curl-8962-ce8d10\" (UID: \"972e51c6-627c-4fe1-9d3c-aa7c89f0b26b\") " pod="opendatahub/test-curl-8962-ce8d10"
Sep 18 18:53:30.825397 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:30.825367    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-752kx\" (UniqueName: \"kubernetes.io/projected/972e51c6-627c-4fe1-9d3c-aa7c89f0b26b-kube-api-access-752kx\") pod \"test-curl-8962-ce8d10\" (UID: \"972e51c6-627c-4fe1-9d3c-aa7c89f0b26b\") " pod="opendatahub/test-curl-8962-ce8d10"
Sep 18 18:53:30.923585 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:30.923501    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8962-ce8d10"
Sep 18 18:53:31.197105 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:31.197076    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8962-ce8d10"]
Sep 18 18:53:31.198687 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:53:31.198650    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod972e51c6_627c_4fe1_9d3c_aa7c89f0b26b.slice/crio-cbb9ffcbd538e2e3bd97d221224503a2f2aaaba321b6c655fa2753fcb06676a9 WatchSource:0}: Error finding container cbb9ffcbd538e2e3bd97d221224503a2f2aaaba321b6c655fa2753fcb06676a9: Status 404 returned error can't find the container with id cbb9ffcbd538e2e3bd97d221224503a2f2aaaba321b6c655fa2753fcb06676a9
Sep 18 18:53:32.090205 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:32.090142    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8962-ce8d10" event={"ID":"972e51c6-627c-4fe1-9d3c-aa7c89f0b26b","Type":"ContainerStarted","Data":"cbb9ffcbd538e2e3bd97d221224503a2f2aaaba321b6c655fa2753fcb06676a9"}
Sep 18 18:53:33.392540 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.392498    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"]
Sep 18 18:53:33.396897 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.396878    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.399687 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.399666    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6cc65f16-openshift-default-dockercfg-j6hx8\""
Sep 18 18:53:33.408174 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.408152    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"]
Sep 18 18:53:33.542784 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.542744    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-podinfo\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.542931 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.542801    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-workload-socket\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.542931 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.542827    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjmds\" (UniqueName: \"kubernetes.io/projected/a188afd9-7495-49f0-9c89-374fd7b3455b-kube-api-access-pjmds\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.542931 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.542877    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-data\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.542931 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.542904    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a188afd9-7495-49f0-9c89-374fd7b3455b-istiod-ca-cert\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.543084 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.543026    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-credential-socket\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.543084 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.543056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-envoy\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.543084 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.543074    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-token\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.543179 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.543139    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-workload-certs\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.644260 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.644154    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-credential-socket\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.644260 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.644230    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-envoy\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.644260 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.644258    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-token\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.644553 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.644308    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-workload-certs\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.644553 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.644399    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-podinfo\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.644553 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.644451    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-workload-socket\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.644553 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.644476    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjmds\" (UniqueName: \"kubernetes.io/projected/a188afd9-7495-49f0-9c89-374fd7b3455b-kube-api-access-pjmds\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.644766 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.644576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-data\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.644766 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.644603    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a188afd9-7495-49f0-9c89-374fd7b3455b-istiod-ca-cert\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.644876 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.644809    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-workload-certs\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.644876 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.644848    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-credential-socket\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.644996 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.644882    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-workload-socket\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.645269 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.645240    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-data\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.645555 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.645538    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a188afd9-7495-49f0-9c89-374fd7b3455b-istiod-ca-cert\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.647027 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.646994    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-envoy\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.647435 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.647415    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-podinfo\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.654071 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.654038    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-token\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.654174 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.654097    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjmds\" (UniqueName: \"kubernetes.io/projected/a188afd9-7495-49f0-9c89-374fd7b3455b-kube-api-access-pjmds\") pod \"e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") " pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:33.710157 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:33.710124    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:34.008414 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:34.008308    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"]
Sep 18 18:53:34.011334 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:53:34.011303    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda188afd9_7495_49f0_9c89_374fd7b3455b.slice/crio-feab5cd2a695f2b18816a512bd54a85fda62eed80097a7ef374c0adfc2cb2c53 WatchSource:0}: Error finding container feab5cd2a695f2b18816a512bd54a85fda62eed80097a7ef374c0adfc2cb2c53: Status 404 returned error can't find the container with id feab5cd2a695f2b18816a512bd54a85fda62eed80097a7ef374c0adfc2cb2c53
Sep 18 18:53:34.013671 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:34.013641    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:34.013743 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:34.013704    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:34.013743 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:34.013732    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:34.107993 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:34.107954    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8962-ce8d10" event={"ID":"972e51c6-627c-4fe1-9d3c-aa7c89f0b26b","Type":"ContainerStarted","Data":"de104f1afec286e68ddb689b03ec9dd185baa35776a5ea701982320dbc882c05"}
Sep 18 18:53:34.109790 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:34.109760    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq" event={"ID":"a188afd9-7495-49f0-9c89-374fd7b3455b","Type":"ContainerStarted","Data":"747b2f940a3ad7e1fc25540c9c919f54d90539eecc3b90170bc49947faded111"}
Sep 18 18:53:34.109911 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:34.109815    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq" event={"ID":"a188afd9-7495-49f0-9c89-374fd7b3455b","Type":"ContainerStarted","Data":"feab5cd2a695f2b18816a512bd54a85fda62eed80097a7ef374c0adfc2cb2c53"}
Sep 18 18:53:34.128645 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:34.128593    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8962-ce8d10" podStartSLOduration=2.28452642 podStartE2EDuration="4.128577128s" totalImagesPullingTime="1.844050708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:53:34.126364995 +0000 UTC m=+873.562943836" watchObservedRunningTime="2026-09-18 18:53:34.128577128 +0000 UTC m=+873.565155969"
Sep 18 18:53:34.159453 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:34.159397    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq" podStartSLOduration=1.159377573 podStartE2EDuration="1.159377573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:53:34.156290508 +0000 UTC m=+873.592869350" watchObservedRunningTime="2026-09-18 18:53:34.159377573 +0000 UTC m=+873.595956416"
Sep 18 18:53:34.710630 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:34.710586    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:34.780325 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:34.780289    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8962-ce8d10"]
Sep 18 18:53:34.785873 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:34.785840    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8962-ce8d10"]
Sep 18 18:53:35.117702 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:53:35.117677    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8962-ce8d10" container="test-curl-8962-ce8d10"
Sep 18 18:53:35.715414 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:35.715380    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:36.123505 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:36.123466    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8962-ce8d10" podUID="972e51c6-627c-4fe1-9d3c-aa7c89f0b26b" containerName="test-curl-8962-ce8d10" containerID="cri-o://de104f1afec286e68ddb689b03ec9dd185baa35776a5ea701982320dbc882c05" gracePeriod=2
Sep 18 18:53:36.123776 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:36.123751    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:36.124914 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:36.124894    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:36.125485 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:53:36.125440    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8962-ce8d10\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="972e51c6-627c-4fe1-9d3c-aa7c89f0b26b" pod="opendatahub/test-curl-8962-ce8d10"
Sep 18 18:53:36.127091 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:53:36.127062    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8962-ce8d10\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="972e51c6-627c-4fe1-9d3c-aa7c89f0b26b" pod="opendatahub/test-curl-8962-ce8d10"
Sep 18 18:53:38.436223 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:38.436198    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8962-ce8d10"
Sep 18 18:53:38.617370 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:38.617338    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-752kx\" (UniqueName: \"kubernetes.io/projected/972e51c6-627c-4fe1-9d3c-aa7c89f0b26b-kube-api-access-752kx\") pod \"972e51c6-627c-4fe1-9d3c-aa7c89f0b26b\" (UID: \"972e51c6-627c-4fe1-9d3c-aa7c89f0b26b\") "
Sep 18 18:53:38.619994 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:38.619957    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972e51c6-627c-4fe1-9d3c-aa7c89f0b26b-kube-api-access-752kx" (OuterVolumeSpecName: "kube-api-access-752kx") pod "972e51c6-627c-4fe1-9d3c-aa7c89f0b26b" (UID: "972e51c6-627c-4fe1-9d3c-aa7c89f0b26b"). InnerVolumeSpecName "kube-api-access-752kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:53:38.718300 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:38.718214    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-752kx\" (UniqueName: \"kubernetes.io/projected/972e51c6-627c-4fe1-9d3c-aa7c89f0b26b-kube-api-access-752kx\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:53:39.013787 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.013704    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"]
Sep 18 18:53:39.014170 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.014156    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972e51c6-627c-4fe1-9d3c-aa7c89f0b26b" containerName="test-curl-8962-ce8d10"
Sep 18 18:53:39.014244 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.014171    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="972e51c6-627c-4fe1-9d3c-aa7c89f0b26b" containerName="test-curl-8962-ce8d10"
Sep 18 18:53:39.014286 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.014261    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="972e51c6-627c-4fe1-9d3c-aa7c89f0b26b" containerName="test-curl-8962-ce8d10"
Sep 18 18:53:39.017482 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.017463    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"
Sep 18 18:53:39.020297 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.020276    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6cc65f16\""
Sep 18 18:53:39.039296 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.039268    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"]
Sep 18 18:53:39.122945 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.122906    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnb2x\" (UniqueName: \"kubernetes.io/projected/26248654-a70a-4d9c-85fc-c9851c4e911e-kube-api-access-mnb2x\") pod \"payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7\" (UID: \"26248654-a70a-4d9c-85fc-c9851c4e911e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"
Sep 18 18:53:39.122945 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.122947    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26248654-a70a-4d9c-85fc-c9851c4e911e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7\" (UID: \"26248654-a70a-4d9c-85fc-c9851c4e911e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"
Sep 18 18:53:39.145380 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.145349    2558 generic.go:356] "Generic (PLEG): container finished" podID="972e51c6-627c-4fe1-9d3c-aa7c89f0b26b" containerID="de104f1afec286e68ddb689b03ec9dd185baa35776a5ea701982320dbc882c05" exitCode=137
Sep 18 18:53:39.145545 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.145400    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8962-ce8d10"
Sep 18 18:53:39.145545 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.145462    2558 scope.go:117] "RemoveContainer" containerID="de104f1afec286e68ddb689b03ec9dd185baa35776a5ea701982320dbc882c05"
Sep 18 18:53:39.210704 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.210667    2558 scope.go:117] "RemoveContainer" containerID="de104f1afec286e68ddb689b03ec9dd185baa35776a5ea701982320dbc882c05"
Sep 18 18:53:39.211100 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:53:39.211077    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de104f1afec286e68ddb689b03ec9dd185baa35776a5ea701982320dbc882c05\": container with ID starting with de104f1afec286e68ddb689b03ec9dd185baa35776a5ea701982320dbc882c05 not found: ID does not exist" containerID="de104f1afec286e68ddb689b03ec9dd185baa35776a5ea701982320dbc882c05"
Sep 18 18:53:39.211153 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.211111    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de104f1afec286e68ddb689b03ec9dd185baa35776a5ea701982320dbc882c05"} err="failed to get container status \"de104f1afec286e68ddb689b03ec9dd185baa35776a5ea701982320dbc882c05\": rpc error: code = NotFound desc = could not find container \"de104f1afec286e68ddb689b03ec9dd185baa35776a5ea701982320dbc882c05\": container with ID starting with de104f1afec286e68ddb689b03ec9dd185baa35776a5ea701982320dbc882c05 not found: ID does not exist"
Sep 18 18:53:39.224332 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.224306    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnb2x\" (UniqueName: \"kubernetes.io/projected/26248654-a70a-4d9c-85fc-c9851c4e911e-kube-api-access-mnb2x\") pod \"payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7\" (UID: \"26248654-a70a-4d9c-85fc-c9851c4e911e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"
Sep 18 18:53:39.224446 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.224336    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26248654-a70a-4d9c-85fc-c9851c4e911e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7\" (UID: \"26248654-a70a-4d9c-85fc-c9851c4e911e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"
Sep 18 18:53:39.224989 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.224970    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26248654-a70a-4d9c-85fc-c9851c4e911e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7\" (UID: \"26248654-a70a-4d9c-85fc-c9851c4e911e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"
Sep 18 18:53:39.233087 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.233065    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnb2x\" (UniqueName: \"kubernetes.io/projected/26248654-a70a-4d9c-85fc-c9851c4e911e-kube-api-access-mnb2x\") pod \"payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7\" (UID: \"26248654-a70a-4d9c-85fc-c9851c4e911e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"
Sep 18 18:53:39.327377 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.327284    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"
Sep 18 18:53:39.659431 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:39.659389    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"]
Sep 18 18:53:39.680983 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:53:39.680935    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26248654_a70a_4d9c_85fc_c9851c4e911e.slice/crio-ed3d05aaa0426be279f17d3cdb804dc9bb7dd80426372d6668e65f18ffaf66cc WatchSource:0}: Error finding container ed3d05aaa0426be279f17d3cdb804dc9bb7dd80426372d6668e65f18ffaf66cc: Status 404 returned error can't find the container with id ed3d05aaa0426be279f17d3cdb804dc9bb7dd80426372d6668e65f18ffaf66cc
Sep 18 18:53:40.186367 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:40.186330    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7" event={"ID":"26248654-a70a-4d9c-85fc-c9851c4e911e","Type":"ContainerStarted","Data":"30f3c245bd6d8a2fb0901adcfbab9a66771823edced78507482f10c8440607c0"}
Sep 18 18:53:40.186621 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:40.186599    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7" event={"ID":"26248654-a70a-4d9c-85fc-c9851c4e911e","Type":"ContainerStarted","Data":"ed3d05aaa0426be279f17d3cdb804dc9bb7dd80426372d6668e65f18ffaf66cc"}
Sep 18 18:53:40.187573 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:40.187554    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"
Sep 18 18:53:40.580443 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:40.580350    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="972e51c6-627c-4fe1-9d3c-aa7c89f0b26b" path="/var/lib/kubelet/pods/972e51c6-627c-4fe1-9d3c-aa7c89f0b26b/volumes"
Sep 18 18:53:43.107390 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:43.107340    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 18 18:53:43.107802 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:43.107429    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj" podUID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 18:53:45.224031 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:45.223953    2558 generic.go:356] "Generic (PLEG): container finished" podID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerID="2844f9c5b2702140a5cd8766edf6a93dbe0836d9351485f62b573bd779489c5d" exitCode=0
Sep 18 18:53:45.224428 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:45.224023    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj" event={"ID":"42bf6cd3-467b-4c9f-9960-c1214c476a44","Type":"ContainerDied","Data":"2844f9c5b2702140a5cd8766edf6a93dbe0836d9351485f62b573bd779489c5d"}
Sep 18 18:53:45.224471 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:45.224458    2558 scope.go:117] "RemoveContainer" containerID="2844f9c5b2702140a5cd8766edf6a93dbe0836d9351485f62b573bd779489c5d"
Sep 18 18:53:45.225533 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:45.225503    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:45.225604 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:45.225576    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:45.225643 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:45.225611    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:45.245413 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:45.245353    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7" podStartSLOduration=7.245337925 podStartE2EDuration="7.245337925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:53:40.210838156 +0000 UTC m=+879.647416999" watchObservedRunningTime="2026-09-18 18:53:45.245337925 +0000 UTC m=+884.681916817"
Sep 18 18:53:46.232761 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:46.231916    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj" event={"ID":"42bf6cd3-467b-4c9f-9960-c1214c476a44","Type":"ContainerStarted","Data":"8438cbb341fcafeca97bb8175c5957e0f8824fcff1f1f12ed4a8c21478e7fd13"}
Sep 18 18:53:46.234969 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:46.234346    2558 generic.go:356] "Generic (PLEG): container finished" podID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerID="b04bde102f863f93aaf8816675e756f2ab29323de7f709896b5e8e3025f1baac" exitCode=0
Sep 18 18:53:46.234969 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:46.234386    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z" event={"ID":"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab","Type":"ContainerDied","Data":"b04bde102f863f93aaf8816675e756f2ab29323de7f709896b5e8e3025f1baac"}
Sep 18 18:53:46.234969 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:46.234699    2558 scope.go:117] "RemoveContainer" containerID="b04bde102f863f93aaf8816675e756f2ab29323de7f709896b5e8e3025f1baac"
Sep 18 18:53:46.236014 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:46.235778    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:46.236014 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:46.235845    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:46.236014 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:46.235877    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:46.517029 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:46.516937    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:46.517179 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:46.517063    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 18 18:53:46.517179 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:46.517115    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj" podUID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 18 18:53:47.243309 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:47.243270    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z" event={"ID":"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab","Type":"ContainerStarted","Data":"0791f34afcb43692b05cbe9b12b91fc1c4e4e109aafc1f80f1d0f3dde3da4ada"}
Sep 18 18:53:47.313907 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:47.313861    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:47.314119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:47.313933    2558 patch_prober.go:28] interesting pod/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 18 18:53:47.314119 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:47.313979    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z" podUID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 18 18:53:47.517361 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:47.517251    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 18 18:53:47.517361 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:47.517326    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj" podUID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 18 18:53:48.312813 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:48.312767    2558 patch_prober.go:28] interesting pod/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 18 18:53:48.313473 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:48.313439    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z" podUID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 18 18:53:48.517355 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:48.517315    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 18 18:53:48.517657 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:48.517628    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj" podUID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 18 18:53:49.312945 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:49.312901    2558 patch_prober.go:28] interesting pod/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 18 18:53:49.313592 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:49.313556    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z" podUID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 18 18:53:49.516437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:49.516395    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 18 18:53:49.516726 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:49.516695    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj" podUID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 18 18:53:50.312848 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:50.312809    2558 patch_prober.go:28] interesting pod/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 18 18:53:50.313174 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:50.313135    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z" podUID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 18 18:53:50.516650 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:50.516603    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 18 18:53:50.516865 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:50.516675    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj" podUID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 18 18:53:51.313303 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:51.313260    2558 patch_prober.go:28] interesting pod/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 18 18:53:51.313810 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:51.313355    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z" podUID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 18 18:53:51.517244 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:51.517198    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 18 18:53:51.517419 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:51.517258    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj" podUID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 18 18:53:52.205260 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:52.205224    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"
Sep 18 18:53:52.313305 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:52.313273    2558 patch_prober.go:28] interesting pod/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 18 18:53:52.313488 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:52.313327    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z" podUID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 18 18:53:52.521735 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:52.521649    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:52.522057 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:52.522028    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:52.522981 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:52.522957    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 18:53:53.318129 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:53.318091    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:53.318582 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:53.318558    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:53.319221 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:53.319176    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 18:53:55.307620 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:55.307584    2558 generic.go:356] "Generic (PLEG): container finished" podID="a188afd9-7495-49f0-9c89-374fd7b3455b" containerID="747b2f940a3ad7e1fc25540c9c919f54d90539eecc3b90170bc49947faded111" exitCode=0
Sep 18 18:53:55.308037 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:55.307652    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq" event={"ID":"a188afd9-7495-49f0-9c89-374fd7b3455b","Type":"ContainerDied","Data":"747b2f940a3ad7e1fc25540c9c919f54d90539eecc3b90170bc49947faded111"}
Sep 18 18:53:55.308086 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:55.308076    2558 scope.go:117] "RemoveContainer" containerID="747b2f940a3ad7e1fc25540c9c919f54d90539eecc3b90170bc49947faded111"
Sep 18 18:53:55.309296 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:55.309261    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:55.309383 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:55.309359    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:55.309422 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:55.309410    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:55.920575 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:55.920537    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s"]
Sep 18 18:53:55.924382 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:55.924364    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s"
Sep 18 18:53:55.926513 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:55.926494    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 18:53:55.926632 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:55.926614    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 18:53:55.931548 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:55.931521    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s"]
Sep 18 18:53:56.053618 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:56.053567    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb1485c9-2686-405f-bf9e-527b5a1585f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s\" (UID: \"bb1485c9-2686-405f-bf9e-527b5a1585f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s"
Sep 18 18:53:56.155017 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:56.154960    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb1485c9-2686-405f-bf9e-527b5a1585f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s\" (UID: \"bb1485c9-2686-405f-bf9e-527b5a1585f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s"
Sep 18 18:53:56.155971 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:56.155925    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb1485c9-2686-405f-bf9e-527b5a1585f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s\" (UID: \"bb1485c9-2686-405f-bf9e-527b5a1585f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s"
Sep 18 18:53:56.236235 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:56.236134    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s"
Sep 18 18:53:56.345786 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:56.345746    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq" event={"ID":"a188afd9-7495-49f0-9c89-374fd7b3455b","Type":"ContainerStarted","Data":"3ac1aa2b382086f98f7a16d8fa5831465acc3a8c72fda34b6e6d775c3cceb581"}
Sep 18 18:53:56.532785 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:56.532759    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s"]
Sep 18 18:53:56.534915 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:53:56.534882    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1485c9_2686_405f_bf9e_527b5a1585f7.slice/crio-b708cff81b4fe1579c7ce7a2775b65278a2b3ba19563ae86fe0ea1a750b90690 WatchSource:0}: Error finding container b708cff81b4fe1579c7ce7a2775b65278a2b3ba19563ae86fe0ea1a750b90690: Status 404 returned error can't find the container with id b708cff81b4fe1579c7ce7a2775b65278a2b3ba19563ae86fe0ea1a750b90690
Sep 18 18:53:56.711272 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:56.711234    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:53:56.711465 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:56.711442    2558 patch_prober.go:28] interesting pod/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 18 18:53:56.711522 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:56.711494    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq" podUID="a188afd9-7495-49f0-9c89-374fd7b3455b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 18 18:53:57.357476 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:57.357436    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s" event={"ID":"bb1485c9-2686-405f-bf9e-527b5a1585f7","Type":"ContainerStarted","Data":"b708cff81b4fe1579c7ce7a2775b65278a2b3ba19563ae86fe0ea1a750b90690"}
Sep 18 18:53:57.711536 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:57.711458    2558 patch_prober.go:28] interesting pod/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 18 18:53:57.711679 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:57.711528    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq" podUID="a188afd9-7495-49f0-9c89-374fd7b3455b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 18 18:53:58.592591 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.592562    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"]
Sep 18 18:53:58.598124 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.598100    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.600597 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.600576    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-897f61-openshift-default-dockercfg-w5fz7\""
Sep 18 18:53:58.609816 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.609789    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"]
Sep 18 18:53:58.696721 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.696364    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istiod-ca-cert\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.696721 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.696412    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-workload-certs\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.696721 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.696505    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-podinfo\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.696721 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.696550    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-credential-socket\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.696721 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.696585    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-workload-socket\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.696721 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.696672    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh8c9\" (UniqueName: \"kubernetes.io/projected/f6d074b8-f8ae-4e64-b200-ad871d0d9473-kube-api-access-jh8c9\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.697246 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.696784    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-token\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.697246 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.696812    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-data\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.697246 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.696861    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-envoy\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.710583 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.710546    2558 patch_prober.go:28] interesting pod/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 18 18:53:58.710779 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.710640    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq" podUID="a188afd9-7495-49f0-9c89-374fd7b3455b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 18 18:53:58.797701 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.797623    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istiod-ca-cert\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.797701 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.797684    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-workload-certs\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.797962 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.797729    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-podinfo\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.797962 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.797760    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-credential-socket\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.797962 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.797799    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-workload-socket\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.797962 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.797863    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh8c9\" (UniqueName: \"kubernetes.io/projected/f6d074b8-f8ae-4e64-b200-ad871d0d9473-kube-api-access-jh8c9\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.797962 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.797950    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-token\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.798303 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.797981    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-data\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.798303 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.798036    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-envoy\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.798756 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.798727    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istiod-ca-cert\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.798893 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.798787    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-workload-certs\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.798959 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.798920    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-workload-socket\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.799030 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.798950    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-credential-socket\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.799165 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.799135    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-data\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.800894 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.800873    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-podinfo\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.801397 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.801377    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-envoy\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.809645 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.809624    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh8c9\" (UniqueName: \"kubernetes.io/projected/f6d074b8-f8ae-4e64-b200-ad871d0d9473-kube-api-access-jh8c9\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.811280 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.810583    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-token\") pod \"e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") " pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:58.914089 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:58.913936    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:53:59.710586 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:59.710552    2558 patch_prober.go:28] interesting pod/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 18 18:53:59.710961 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:59.710608    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq" podUID="a188afd9-7495-49f0-9c89-374fd7b3455b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 18 18:53:59.802295 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:59.802264    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"]
Sep 18 18:53:59.803651 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:53:59.803625    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d074b8_f8ae_4e64_b200_ad871d0d9473.slice/crio-f650a3c84629f96defd1b1d5c077ab37dce222c20f0db66b5e6252a5cd7a4518 WatchSource:0}: Error finding container f650a3c84629f96defd1b1d5c077ab37dce222c20f0db66b5e6252a5cd7a4518: Status 404 returned error can't find the container with id f650a3c84629f96defd1b1d5c077ab37dce222c20f0db66b5e6252a5cd7a4518
Sep 18 18:53:59.806242 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:59.806176    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:59.806322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:59.806276    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:53:59.806322 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:53:59.806306    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:54:00.392042 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:00.392002    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch" event={"ID":"f6d074b8-f8ae-4e64-b200-ad871d0d9473","Type":"ContainerStarted","Data":"999c7a4d40436df6a06419a2cb28f046ed510269652f1fcf356c72b130765cf8"}
Sep 18 18:54:00.392254 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:00.392068    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch" event={"ID":"f6d074b8-f8ae-4e64-b200-ad871d0d9473","Type":"ContainerStarted","Data":"f650a3c84629f96defd1b1d5c077ab37dce222c20f0db66b5e6252a5cd7a4518"}
Sep 18 18:54:00.393765 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:00.393740    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s" event={"ID":"bb1485c9-2686-405f-bf9e-527b5a1585f7","Type":"ContainerStarted","Data":"515f6c10bf18541e6fc521124458e4e7f5d87f2400d053103b26d40f01936164"}
Sep 18 18:54:00.413257 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:00.413212    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch" podStartSLOduration=2.41317955 podStartE2EDuration="2.41317955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:54:00.411082424 +0000 UTC m=+899.847661265" watchObservedRunningTime="2026-09-18 18:54:00.41317955 +0000 UTC m=+899.849758392"
Sep 18 18:54:00.426749 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:00.426704    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s" podStartSLOduration=2.358810792 podStartE2EDuration="5.426690886s" totalImagesPullingTime="3.067880094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:54:00.4253293 +0000 UTC m=+899.861908154" watchObservedRunningTime="2026-09-18 18:54:00.426690886 +0000 UTC m=+899.863269726"
Sep 18 18:54:00.710737 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:00.710647    2558 patch_prober.go:28] interesting pod/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 18 18:54:00.710737 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:00.710715    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq" podUID="a188afd9-7495-49f0-9c89-374fd7b3455b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 18 18:54:00.914584 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:00.914547    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:54:00.920168 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:00.920143    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:54:01.401576 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:01.401544    2558 generic.go:356] "Generic (PLEG): container finished" podID="bb1485c9-2686-405f-bf9e-527b5a1585f7" containerID="515f6c10bf18541e6fc521124458e4e7f5d87f2400d053103b26d40f01936164" exitCode=0
Sep 18 18:54:01.401773 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:01.401627    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s" event={"ID":"bb1485c9-2686-405f-bf9e-527b5a1585f7","Type":"ContainerDied","Data":"515f6c10bf18541e6fc521124458e4e7f5d87f2400d053103b26d40f01936164"}
Sep 18 18:54:01.401948 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:01.401933    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:54:01.402950 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:01.402930    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:54:01.715268 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:01.715149    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:54:01.715637 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:01.715457    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:54:01.716115 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:01.716099    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:54:02.545530 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:02.545506    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s"
Sep 18 18:54:02.608520 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:02.608486    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 18:54:02.609581 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:02.609559    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 18:54:02.611103 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:02.611083    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 18:54:02.612364 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:02.612344    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 18:54:02.636870 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:02.636848    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb1485c9-2686-405f-bf9e-527b5a1585f7-maas-api-service-ca\") pod \"bb1485c9-2686-405f-bf9e-527b5a1585f7\" (UID: \"bb1485c9-2686-405f-bf9e-527b5a1585f7\") "
Sep 18 18:54:02.637144 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:02.637124    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb1485c9-2686-405f-bf9e-527b5a1585f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb1485c9-2686-405f-bf9e-527b5a1585f7" (UID: "bb1485c9-2686-405f-bf9e-527b5a1585f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:54:02.738132 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:02.738046    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb1485c9-2686-405f-bf9e-527b5a1585f7-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:03.415579 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:03.415555    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s"
Sep 18 18:54:03.415734 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:03.415600    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s" event={"ID":"bb1485c9-2686-405f-bf9e-527b5a1585f7","Type":"ContainerDied","Data":"b708cff81b4fe1579c7ce7a2775b65278a2b3ba19563ae86fe0ea1a750b90690"}
Sep 18 18:54:03.415734 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:03.415647    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b708cff81b4fe1579c7ce7a2775b65278a2b3ba19563ae86fe0ea1a750b90690"
Sep 18 18:54:07.433203 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.433155    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"]
Sep 18 18:54:07.433639 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.433623    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb1485c9-2686-405f-bf9e-527b5a1585f7" containerName="revoke-keys"
Sep 18 18:54:07.433680 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.433640    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb1485c9-2686-405f-bf9e-527b5a1585f7" containerName="revoke-keys"
Sep 18 18:54:07.433728 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.433709    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb1485c9-2686-405f-bf9e-527b5a1585f7" containerName="revoke-keys"
Sep 18 18:54:07.440819 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.440792    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.444359 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.444332    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9d9f36\"/\"openshift-service-ca.crt\""
Sep 18 18:54:07.444507 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.444365    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9d9f36\"/\"default-dockercfg-2khll\""
Sep 18 18:54:07.444643 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.444622    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9d9f36\"/\"kube-root-ca.crt\""
Sep 18 18:54:07.444769 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.444747    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9d9f36\"/\"simulator-w3-9d9f36-kserve-self-signed-certs\""
Sep 18 18:54:07.453069 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.452214    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"]
Sep 18 18:54:07.490353 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.490325    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxxs\" (UniqueName: \"kubernetes.io/projected/2cfa6716-3dda-4774-b7ae-9339f0657634-kube-api-access-dpxxs\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.490495 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.490369    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-model-cache\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.490495 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.490404    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-tmp-dir\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.490495 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.490438    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-dshm\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.490495 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.490479    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cfa6716-3dda-4774-b7ae-9339f0657634-tls-certs\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.490660 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.490508    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-home\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.527812 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.527783    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"]
Sep 18 18:54:07.528016 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.527985    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7" podUID="26248654-a70a-4d9c-85fc-c9851c4e911e" containerName="payload-pre-processing" containerID="cri-o://30f3c245bd6d8a2fb0901adcfbab9a66771823edced78507482f10c8440607c0" gracePeriod=30
Sep 18 18:54:07.591060 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.591028    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-model-cache\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.591233 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.591066    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-tmp-dir\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.591233 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.591086    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-dshm\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.591331 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.591260    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cfa6716-3dda-4774-b7ae-9339f0657634-tls-certs\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.591385 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.591352    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-home\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.591459 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.591440    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxxs\" (UniqueName: \"kubernetes.io/projected/2cfa6716-3dda-4774-b7ae-9339f0657634-kube-api-access-dpxxs\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.591521 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.591467    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-model-cache\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.591586 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.591515    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-tmp-dir\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.591818 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.591795    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-home\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.593397 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.593377    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-dshm\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.593677 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.593662    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cfa6716-3dda-4774-b7ae-9339f0657634-tls-certs\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.599453 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.599432    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxxs\" (UniqueName: \"kubernetes.io/projected/2cfa6716-3dda-4774-b7ae-9339f0657634-kube-api-access-dpxxs\") pod \"simulator-w3-9d9f36-kserve-769969484f-7sgtl\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") " pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.707879 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.707848    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s"]
Sep 18 18:54:07.713399 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.713370    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6cc65f16-jk56s"]
Sep 18 18:54:07.756999 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.756966    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:07.796378 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.796347    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"
Sep 18 18:54:07.836751 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.836657    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"]
Sep 18 18:54:07.837341 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.837137    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq" podUID="a188afd9-7495-49f0-9c89-374fd7b3455b" containerName="istio-proxy" containerID="cri-o://3ac1aa2b382086f98f7a16d8fa5831465acc3a8c72fda34b6e6d775c3cceb581" gracePeriod=30
Sep 18 18:54:07.893920 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.893893    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnb2x\" (UniqueName: \"kubernetes.io/projected/26248654-a70a-4d9c-85fc-c9851c4e911e-kube-api-access-mnb2x\") pod \"26248654-a70a-4d9c-85fc-c9851c4e911e\" (UID: \"26248654-a70a-4d9c-85fc-c9851c4e911e\") "
Sep 18 18:54:07.894102 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.893976    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26248654-a70a-4d9c-85fc-c9851c4e911e-plugins-config-volume\") pod \"26248654-a70a-4d9c-85fc-c9851c4e911e\" (UID: \"26248654-a70a-4d9c-85fc-c9851c4e911e\") "
Sep 18 18:54:07.894412 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.894392    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26248654-a70a-4d9c-85fc-c9851c4e911e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26248654-a70a-4d9c-85fc-c9851c4e911e" (UID: "26248654-a70a-4d9c-85fc-c9851c4e911e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:54:07.896002 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.895981    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26248654-a70a-4d9c-85fc-c9851c4e911e-kube-api-access-mnb2x" (OuterVolumeSpecName: "kube-api-access-mnb2x") pod "26248654-a70a-4d9c-85fc-c9851c4e911e" (UID: "26248654-a70a-4d9c-85fc-c9851c4e911e"). InnerVolumeSpecName "kube-api-access-mnb2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:54:07.995739 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.995642    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnb2x\" (UniqueName: \"kubernetes.io/projected/26248654-a70a-4d9c-85fc-c9851c4e911e-kube-api-access-mnb2x\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:07.995739 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:07.995674    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26248654-a70a-4d9c-85fc-c9851c4e911e-plugins-config-volume\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:08.041534 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.041508    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"]
Sep 18 18:54:08.042768 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:54:08.042740    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cfa6716_3dda_4774_b7ae_9339f0657634.slice/crio-dd96338167e67aeb5833d9cc1b7c2fbdc798f77f3128944fb454a716715a4437 WatchSource:0}: Error finding container dd96338167e67aeb5833d9cc1b7c2fbdc798f77f3128944fb454a716715a4437: Status 404 returned error can't find the container with id dd96338167e67aeb5833d9cc1b7c2fbdc798f77f3128944fb454a716715a4437
Sep 18 18:54:08.449380 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.449350    2558 generic.go:356] "Generic (PLEG): container finished" podID="26248654-a70a-4d9c-85fc-c9851c4e911e" containerID="30f3c245bd6d8a2fb0901adcfbab9a66771823edced78507482f10c8440607c0" exitCode=0
Sep 18 18:54:08.449835 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.449409    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"
Sep 18 18:54:08.449835 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.449440    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7" event={"ID":"26248654-a70a-4d9c-85fc-c9851c4e911e","Type":"ContainerDied","Data":"30f3c245bd6d8a2fb0901adcfbab9a66771823edced78507482f10c8440607c0"}
Sep 18 18:54:08.449835 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.449498    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7" event={"ID":"26248654-a70a-4d9c-85fc-c9851c4e911e","Type":"ContainerDied","Data":"ed3d05aaa0426be279f17d3cdb804dc9bb7dd80426372d6668e65f18ffaf66cc"}
Sep 18 18:54:08.449835 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.449532    2558 scope.go:117] "RemoveContainer" containerID="30f3c245bd6d8a2fb0901adcfbab9a66771823edced78507482f10c8440607c0"
Sep 18 18:54:08.451264 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.451175    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl" event={"ID":"2cfa6716-3dda-4774-b7ae-9339f0657634","Type":"ContainerStarted","Data":"c07a17fb1a4f409989bcae283c7779b058759be398e387e4ca47f3442a19b92b"}
Sep 18 18:54:08.451264 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.451240    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl" event={"ID":"2cfa6716-3dda-4774-b7ae-9339f0657634","Type":"ContainerStarted","Data":"dd96338167e67aeb5833d9cc1b7c2fbdc798f77f3128944fb454a716715a4437"}
Sep 18 18:54:08.467039 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.467019    2558 scope.go:117] "RemoveContainer" containerID="30f3c245bd6d8a2fb0901adcfbab9a66771823edced78507482f10c8440607c0"
Sep 18 18:54:08.467354 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:54:08.467333    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f3c245bd6d8a2fb0901adcfbab9a66771823edced78507482f10c8440607c0\": container with ID starting with 30f3c245bd6d8a2fb0901adcfbab9a66771823edced78507482f10c8440607c0 not found: ID does not exist" containerID="30f3c245bd6d8a2fb0901adcfbab9a66771823edced78507482f10c8440607c0"
Sep 18 18:54:08.467437 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.467360    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f3c245bd6d8a2fb0901adcfbab9a66771823edced78507482f10c8440607c0"} err="failed to get container status \"30f3c245bd6d8a2fb0901adcfbab9a66771823edced78507482f10c8440607c0\": rpc error: code = NotFound desc = could not find container \"30f3c245bd6d8a2fb0901adcfbab9a66771823edced78507482f10c8440607c0\": container with ID starting with 30f3c245bd6d8a2fb0901adcfbab9a66771823edced78507482f10c8440607c0 not found: ID does not exist"
Sep 18 18:54:08.476332 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.476292    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl" podStartSLOduration=1.476279716 podStartE2EDuration="1.476279716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:54:08.474983765 +0000 UTC m=+907.911562598" watchObservedRunningTime="2026-09-18 18:54:08.476279716 +0000 UTC m=+907.912858599"
Sep 18 18:54:08.492606 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.492581    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"]
Sep 18 18:54:08.499033 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.499007    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6cc65f16-696f658d98-zpkq7"]
Sep 18 18:54:08.575875 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.575843    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26248654-a70a-4d9c-85fc-c9851c4e911e" path="/var/lib/kubelet/pods/26248654-a70a-4d9c-85fc-c9851c4e911e/volumes"
Sep 18 18:54:08.576324 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.576303    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb1485c9-2686-405f-bf9e-527b5a1585f7" path="/var/lib/kubelet/pods/bb1485c9-2686-405f-bf9e-527b5a1585f7/volumes"
Sep 18 18:54:08.755102 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.755013    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"]
Sep 18 18:54:08.755489 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.755472    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26248654-a70a-4d9c-85fc-c9851c4e911e" containerName="payload-pre-processing"
Sep 18 18:54:08.755489 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.755488    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="26248654-a70a-4d9c-85fc-c9851c4e911e" containerName="payload-pre-processing"
Sep 18 18:54:08.755642 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.755578    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="26248654-a70a-4d9c-85fc-c9851c4e911e" containerName="payload-pre-processing"
Sep 18 18:54:08.760053 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.760025    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.762383 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.762363    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e8c645ee-gw-openshift-default-dockercfg-bzv59\""
Sep 18 18:54:08.772030 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.772006    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"]
Sep 18 18:54:08.804761 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.804690    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-credential-socket\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.804761 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.804735    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-envoy\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.804977 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.804882    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-workload-socket\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.804977 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.804909    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46118a5a-9491-4cf0-a2a2-84e76137516a-istiod-ca-cert\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.804977 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.804962    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmnrx\" (UniqueName: \"kubernetes.io/projected/46118a5a-9491-4cf0-a2a2-84e76137516a-kube-api-access-qmnrx\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.805499 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.805031    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-podinfo\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.805499 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.805055    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-data\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.805499 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.805108    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-workload-certs\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.805499 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.805360    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-token\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.906075 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.906036    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-workload-socket\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.906075 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.906076    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46118a5a-9491-4cf0-a2a2-84e76137516a-istiod-ca-cert\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.906320 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.906231    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnrx\" (UniqueName: \"kubernetes.io/projected/46118a5a-9491-4cf0-a2a2-84e76137516a-kube-api-access-qmnrx\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.906392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.906329    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-podinfo\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.906392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.906361    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-data\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.906496 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.906395    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-workload-certs\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.906496 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.906461    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-token\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.906654 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.906508    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-credential-socket\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.906654 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.906523    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-workload-socket\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.906654 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.906539    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-envoy\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.906822 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.906765    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-data\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.906878 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.906866    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46118a5a-9491-4cf0-a2a2-84e76137516a-istiod-ca-cert\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.907116 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.907079    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-workload-certs\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.907293 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.907272    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-credential-socket\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.908866 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.908837    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-envoy\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.909060 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.909040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-podinfo\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.917713 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.917655    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnrx\" (UniqueName: \"kubernetes.io/projected/46118a5a-9491-4cf0-a2a2-84e76137516a-kube-api-access-qmnrx\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:08.932217 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:08.928158    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-token\") pod \"e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") " pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:09.073645 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:09.073564    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:09.356061 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:09.356033    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"]
Sep 18 18:54:09.357362 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:54:09.357337    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46118a5a_9491_4cf0_a2a2_84e76137516a.slice/crio-47458d422b3218094c0e4506c300d9d02c6b997ae25e651a8e18076c73981fca WatchSource:0}: Error finding container 47458d422b3218094c0e4506c300d9d02c6b997ae25e651a8e18076c73981fca: Status 404 returned error can't find the container with id 47458d422b3218094c0e4506c300d9d02c6b997ae25e651a8e18076c73981fca
Sep 18 18:54:09.360230 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:09.360174    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:54:09.360320 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:09.360273    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:54:09.360320 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:09.360301    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:54:09.461904 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:09.461864    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx" event={"ID":"46118a5a-9491-4cf0-a2a2-84e76137516a","Type":"ContainerStarted","Data":"b1115ae7fe3d235489900c62f2268c07a1776195f32693d8768dbf83b470c564"}
Sep 18 18:54:09.462319 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:09.461935    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx" event={"ID":"46118a5a-9491-4cf0-a2a2-84e76137516a","Type":"ContainerStarted","Data":"47458d422b3218094c0e4506c300d9d02c6b997ae25e651a8e18076c73981fca"}
Sep 18 18:54:09.486041 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:09.485984    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx" podStartSLOduration=1.485963001 podStartE2EDuration="1.485963001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:54:09.485495289 +0000 UTC m=+908.922074142" watchObservedRunningTime="2026-09-18 18:54:09.485963001 +0000 UTC m=+908.922541844"
Sep 18 18:54:10.074041 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:10.074002    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:10.079137 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:10.079112    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:10.468141 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:10.468112    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:10.469122 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:10.469102    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:13.089986 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.089961    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:54:13.151326 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.151245    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a188afd9-7495-49f0-9c89-374fd7b3455b-istiod-ca-cert\") pod \"a188afd9-7495-49f0-9c89-374fd7b3455b\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") "
Sep 18 18:54:13.151326 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.151297    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-workload-certs\") pod \"a188afd9-7495-49f0-9c89-374fd7b3455b\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") "
Sep 18 18:54:13.151542 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.151337    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-token\") pod \"a188afd9-7495-49f0-9c89-374fd7b3455b\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") "
Sep 18 18:54:13.151542 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.151367    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-workload-socket\") pod \"a188afd9-7495-49f0-9c89-374fd7b3455b\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") "
Sep 18 18:54:13.151542 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.151397    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-podinfo\") pod \"a188afd9-7495-49f0-9c89-374fd7b3455b\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") "
Sep 18 18:54:13.151542 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.151438    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjmds\" (UniqueName: \"kubernetes.io/projected/a188afd9-7495-49f0-9c89-374fd7b3455b-kube-api-access-pjmds\") pod \"a188afd9-7495-49f0-9c89-374fd7b3455b\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") "
Sep 18 18:54:13.151542 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.151490    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-data\") pod \"a188afd9-7495-49f0-9c89-374fd7b3455b\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") "
Sep 18 18:54:13.151542 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.151536    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-envoy\") pod \"a188afd9-7495-49f0-9c89-374fd7b3455b\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") "
Sep 18 18:54:13.151904 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.151573    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-credential-socket\") pod \"a188afd9-7495-49f0-9c89-374fd7b3455b\" (UID: \"a188afd9-7495-49f0-9c89-374fd7b3455b\") "
Sep 18 18:54:13.151904 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.151626    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a188afd9-7495-49f0-9c89-374fd7b3455b" (UID: "a188afd9-7495-49f0-9c89-374fd7b3455b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:54:13.152010 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.151728    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a188afd9-7495-49f0-9c89-374fd7b3455b" (UID: "a188afd9-7495-49f0-9c89-374fd7b3455b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:54:13.152010 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.151757    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a188afd9-7495-49f0-9c89-374fd7b3455b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a188afd9-7495-49f0-9c89-374fd7b3455b" (UID: "a188afd9-7495-49f0-9c89-374fd7b3455b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:54:13.152010 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.151926    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-data" (OuterVolumeSpecName: "istio-data") pod "a188afd9-7495-49f0-9c89-374fd7b3455b" (UID: "a188afd9-7495-49f0-9c89-374fd7b3455b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:54:13.152159 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.152087    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a188afd9-7495-49f0-9c89-374fd7b3455b" (UID: "a188afd9-7495-49f0-9c89-374fd7b3455b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:54:13.154028 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.153995    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a188afd9-7495-49f0-9c89-374fd7b3455b" (UID: "a188afd9-7495-49f0-9c89-374fd7b3455b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:54:13.154255 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.154230    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a188afd9-7495-49f0-9c89-374fd7b3455b-kube-api-access-pjmds" (OuterVolumeSpecName: "kube-api-access-pjmds") pod "a188afd9-7495-49f0-9c89-374fd7b3455b" (UID: "a188afd9-7495-49f0-9c89-374fd7b3455b"). InnerVolumeSpecName "kube-api-access-pjmds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:54:13.154394 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.154374    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a188afd9-7495-49f0-9c89-374fd7b3455b" (UID: "a188afd9-7495-49f0-9c89-374fd7b3455b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:54:13.154730 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.154708    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-token" (OuterVolumeSpecName: "istio-token") pod "a188afd9-7495-49f0-9c89-374fd7b3455b" (UID: "a188afd9-7495-49f0-9c89-374fd7b3455b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:54:13.253493 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.253454    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a188afd9-7495-49f0-9c89-374fd7b3455b-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:13.253493 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.253491    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:13.253734 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.253505    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:13.253734 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.253520    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:13.253734 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.253532    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:13.253734 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.253548    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjmds\" (UniqueName: \"kubernetes.io/projected/a188afd9-7495-49f0-9c89-374fd7b3455b-kube-api-access-pjmds\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:13.253734 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.253562    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:13.253734 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.253575    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:13.253734 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.253587    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a188afd9-7495-49f0-9c89-374fd7b3455b-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:13.494576 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.494485    2558 generic.go:356] "Generic (PLEG): container finished" podID="a188afd9-7495-49f0-9c89-374fd7b3455b" containerID="3ac1aa2b382086f98f7a16d8fa5831465acc3a8c72fda34b6e6d775c3cceb581" exitCode=0
Sep 18 18:54:13.494576 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.494536    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"
Sep 18 18:54:13.494576 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.494564    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq" event={"ID":"a188afd9-7495-49f0-9c89-374fd7b3455b","Type":"ContainerDied","Data":"3ac1aa2b382086f98f7a16d8fa5831465acc3a8c72fda34b6e6d775c3cceb581"}
Sep 18 18:54:13.494844 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.494623    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq" event={"ID":"a188afd9-7495-49f0-9c89-374fd7b3455b","Type":"ContainerDied","Data":"feab5cd2a695f2b18816a512bd54a85fda62eed80097a7ef374c0adfc2cb2c53"}
Sep 18 18:54:13.494844 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.494652    2558 scope.go:117] "RemoveContainer" containerID="3ac1aa2b382086f98f7a16d8fa5831465acc3a8c72fda34b6e6d775c3cceb581"
Sep 18 18:54:13.505737 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.505505    2558 scope.go:117] "RemoveContainer" containerID="747b2f940a3ad7e1fc25540c9c919f54d90539eecc3b90170bc49947faded111"
Sep 18 18:54:13.515056 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.515038    2558 scope.go:117] "RemoveContainer" containerID="3ac1aa2b382086f98f7a16d8fa5831465acc3a8c72fda34b6e6d775c3cceb581"
Sep 18 18:54:13.515418 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:54:13.515399    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac1aa2b382086f98f7a16d8fa5831465acc3a8c72fda34b6e6d775c3cceb581\": container with ID starting with 3ac1aa2b382086f98f7a16d8fa5831465acc3a8c72fda34b6e6d775c3cceb581 not found: ID does not exist" containerID="3ac1aa2b382086f98f7a16d8fa5831465acc3a8c72fda34b6e6d775c3cceb581"
Sep 18 18:54:13.515482 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.515428    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac1aa2b382086f98f7a16d8fa5831465acc3a8c72fda34b6e6d775c3cceb581"} err="failed to get container status \"3ac1aa2b382086f98f7a16d8fa5831465acc3a8c72fda34b6e6d775c3cceb581\": rpc error: code = NotFound desc = could not find container \"3ac1aa2b382086f98f7a16d8fa5831465acc3a8c72fda34b6e6d775c3cceb581\": container with ID starting with 3ac1aa2b382086f98f7a16d8fa5831465acc3a8c72fda34b6e6d775c3cceb581 not found: ID does not exist"
Sep 18 18:54:13.515482 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.515445    2558 scope.go:117] "RemoveContainer" containerID="747b2f940a3ad7e1fc25540c9c919f54d90539eecc3b90170bc49947faded111"
Sep 18 18:54:13.515720 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:54:13.515689    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747b2f940a3ad7e1fc25540c9c919f54d90539eecc3b90170bc49947faded111\": container with ID starting with 747b2f940a3ad7e1fc25540c9c919f54d90539eecc3b90170bc49947faded111 not found: ID does not exist" containerID="747b2f940a3ad7e1fc25540c9c919f54d90539eecc3b90170bc49947faded111"
Sep 18 18:54:13.515720 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.515724    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747b2f940a3ad7e1fc25540c9c919f54d90539eecc3b90170bc49947faded111"} err="failed to get container status \"747b2f940a3ad7e1fc25540c9c919f54d90539eecc3b90170bc49947faded111\": rpc error: code = NotFound desc = could not find container \"747b2f940a3ad7e1fc25540c9c919f54d90539eecc3b90170bc49947faded111\": container with ID starting with 747b2f940a3ad7e1fc25540c9c919f54d90539eecc3b90170bc49947faded111 not found: ID does not exist"
Sep 18 18:54:13.518052 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.518029    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"]
Sep 18 18:54:13.520600 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:13.520581    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6cc65f16-openshift-default-5ff6f96fdd-rbxmq"]
Sep 18 18:54:14.579307 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:14.579274    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a188afd9-7495-49f0-9c89-374fd7b3455b" path="/var/lib/kubelet/pods/a188afd9-7495-49f0-9c89-374fd7b3455b/volumes"
Sep 18 18:54:17.538036 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.538002    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"]
Sep 18 18:54:17.538463 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.538449    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a188afd9-7495-49f0-9c89-374fd7b3455b" containerName="istio-proxy"
Sep 18 18:54:17.538505 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.538465    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a188afd9-7495-49f0-9c89-374fd7b3455b" containerName="istio-proxy"
Sep 18 18:54:17.538505 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.538476    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a188afd9-7495-49f0-9c89-374fd7b3455b" containerName="istio-proxy"
Sep 18 18:54:17.538505 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.538481    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a188afd9-7495-49f0-9c89-374fd7b3455b" containerName="istio-proxy"
Sep 18 18:54:17.538599 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.538555    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a188afd9-7495-49f0-9c89-374fd7b3455b" containerName="istio-proxy"
Sep 18 18:54:17.543038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.543020    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.550581 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.550556    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-cc289c\"/\"kube-root-ca.crt\""
Sep 18 18:54:17.551019 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.550998    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-cc289c\"/\"simulator-w0-cc289c-kserve-self-signed-certs\""
Sep 18 18:54:17.551366 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.551347    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-cc289c\"/\"openshift-service-ca.crt\""
Sep 18 18:54:17.551709 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.551691    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-cc289c\"/\"default-dockercfg-t552s\""
Sep 18 18:54:17.563256 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.563232    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"]
Sep 18 18:54:17.600673 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.600635    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56ef4d9-cf5b-4f41-9304-4ee529cea356-tls-certs\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.600847 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.600693    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrjfd\" (UniqueName: \"kubernetes.io/projected/c56ef4d9-cf5b-4f41-9304-4ee529cea356-kube-api-access-zrjfd\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.600847 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.600804    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-model-cache\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.600993 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.600848    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-dshm\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.600993 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.600943    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-tmp-dir\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.600993 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.600968    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-home\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.702147 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.702100    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-model-cache\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.702326 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.702153    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-dshm\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.702326 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.702244    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-tmp-dir\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.702326 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.702268    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-home\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.702507 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.702381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56ef4d9-cf5b-4f41-9304-4ee529cea356-tls-certs\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.702507 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.702468    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrjfd\" (UniqueName: \"kubernetes.io/projected/c56ef4d9-cf5b-4f41-9304-4ee529cea356-kube-api-access-zrjfd\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.702612 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.702595    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-model-cache\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.702672 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.702648    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-tmp-dir\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.703064 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.703011    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-home\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.705215 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.705170    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-dshm\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.705530 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.705505    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56ef4d9-cf5b-4f41-9304-4ee529cea356-tls-certs\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.711499 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.711479    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrjfd\" (UniqueName: \"kubernetes.io/projected/c56ef4d9-cf5b-4f41-9304-4ee529cea356-kube-api-access-zrjfd\") pod \"simulator-w0-cc289c-kserve-679db4654f-lpxvl\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") " pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:17.761960 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.759591    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:17.762707 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.762676    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:17.785472 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.785436    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:17.863397 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:17.863293    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:18.153368 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:18.153241    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"]
Sep 18 18:54:18.160696 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:54:18.160662    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc56ef4d9_cf5b_4f41_9304_4ee529cea356.slice/crio-68e0a63f71bd4d79aecf5de772ccc4ba4c088e69df0ea3223d9ef424fe6aa210 WatchSource:0}: Error finding container 68e0a63f71bd4d79aecf5de772ccc4ba4c088e69df0ea3223d9ef424fe6aa210: Status 404 returned error can't find the container with id 68e0a63f71bd4d79aecf5de772ccc4ba4c088e69df0ea3223d9ef424fe6aa210
Sep 18 18:54:18.534217 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:18.534107    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl" event={"ID":"c56ef4d9-cf5b-4f41-9304-4ee529cea356","Type":"ContainerStarted","Data":"a2f9c2fec418db4662221000b8cf9b872bd05f03a02e9a7f89af10c420ca0fa1"}
Sep 18 18:54:18.534217 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:18.534149    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl" event={"ID":"c56ef4d9-cf5b-4f41-9304-4ee529cea356","Type":"ContainerStarted","Data":"68e0a63f71bd4d79aecf5de772ccc4ba4c088e69df0ea3223d9ef424fe6aa210"}
Sep 18 18:54:18.545301 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:18.545278    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 18:54:18.554867 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:18.554812    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl" podStartSLOduration=1.554792666 podStartE2EDuration="1.554792666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:54:18.551482932 +0000 UTC m=+917.988061778" watchObservedRunningTime="2026-09-18 18:54:18.554792666 +0000 UTC m=+917.991371509"
Sep 18 18:54:23.570966 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:23.570935    2558 generic.go:356] "Generic (PLEG): container finished" podID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerID="999c7a4d40436df6a06419a2cb28f046ed510269652f1fcf356c72b130765cf8" exitCode=0
Sep 18 18:54:23.571531 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:23.570998    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch" event={"ID":"f6d074b8-f8ae-4e64-b200-ad871d0d9473","Type":"ContainerDied","Data":"999c7a4d40436df6a06419a2cb28f046ed510269652f1fcf356c72b130765cf8"}
Sep 18 18:54:23.571673 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:23.571654    2558 scope.go:117] "RemoveContainer" containerID="999c7a4d40436df6a06419a2cb28f046ed510269652f1fcf356c72b130765cf8"
Sep 18 18:54:23.573645 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:23.573590    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:54:23.573810 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:23.573682    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:54:23.573810 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:23.573733    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:54:24.581749 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:24.581710    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch" event={"ID":"f6d074b8-f8ae-4e64-b200-ad871d0d9473","Type":"ContainerStarted","Data":"99a40c27ebeab44776884112a4fd42c98949273c227483d0e73919b34933d30c"}
Sep 18 18:54:24.914665 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:24.914626    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:54:24.914872 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:24.914845    2558 patch_prober.go:28] interesting pod/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 18 18:54:24.914932 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:24.914912    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch" podUID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 18 18:54:25.914664 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:25.914626    2558 patch_prober.go:28] interesting pod/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 18 18:54:25.915063 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:25.914690    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch" podUID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 18 18:54:26.915276 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:26.915234    2558 patch_prober.go:28] interesting pod/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 18 18:54:26.915816 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:26.915304    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch" podUID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 18 18:54:27.864934 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:27.864879    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:27.865261 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:27.865242    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:27.909018 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:27.908981    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:27.915060 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:27.915037    2558 patch_prober.go:28] interesting pod/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 18 18:54:27.915164 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:27.915106    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch" podUID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 18 18:54:28.629566 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:28.629532    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 18:54:28.915368 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:28.915271    2558 patch_prober.go:28] interesting pod/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 18 18:54:28.915525 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:28.915367    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch" podUID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 18 18:54:29.914460 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:29.914421    2558 patch_prober.go:28] interesting pod/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 18 18:54:29.914859 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:29.914475    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch" podUID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 18 18:54:30.772744 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:30.772696    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp"]
Sep 18 18:54:30.773538 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:30.773510    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a188afd9-7495-49f0-9c89-374fd7b3455b" containerName="istio-proxy"
Sep 18 18:54:30.781319 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:30.781295    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp"
Sep 18 18:54:30.783695 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:30.783669    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 18:54:30.783834 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:30.783814    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 18:54:30.790149 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:30.789902    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp"]
Sep 18 18:54:30.859822 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:30.859775    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5908722-af08-4f95-8d85-e4ad444092b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp\" (UID: \"d5908722-af08-4f95-8d85-e4ad444092b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp"
Sep 18 18:54:30.918931 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:30.918899    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:54:30.919407 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:30.919207    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:54:30.920050 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:30.920033    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:54:30.962345 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:30.962306    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5908722-af08-4f95-8d85-e4ad444092b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp\" (UID: \"d5908722-af08-4f95-8d85-e4ad444092b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp"
Sep 18 18:54:30.970214 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:30.963463    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5908722-af08-4f95-8d85-e4ad444092b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp\" (UID: \"d5908722-af08-4f95-8d85-e4ad444092b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp"
Sep 18 18:54:31.093787 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:31.093754    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp"
Sep 18 18:54:31.369293 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:31.369266    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp"]
Sep 18 18:54:31.371155 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:54:31.371125    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5908722_af08_4f95_8d85_e4ad444092b4.slice/crio-4af17a1ac48769c1f85b9a55a7868b5c697c2fa021d630baf3626b24fe95d6ac WatchSource:0}: Error finding container 4af17a1ac48769c1f85b9a55a7868b5c697c2fa021d630baf3626b24fe95d6ac: Status 404 returned error can't find the container with id 4af17a1ac48769c1f85b9a55a7868b5c697c2fa021d630baf3626b24fe95d6ac
Sep 18 18:54:31.638630 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:31.638540    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp" event={"ID":"d5908722-af08-4f95-8d85-e4ad444092b4","Type":"ContainerStarted","Data":"4af17a1ac48769c1f85b9a55a7868b5c697c2fa021d630baf3626b24fe95d6ac"}
Sep 18 18:54:32.648546 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:32.648508    2558 generic.go:356] "Generic (PLEG): container finished" podID="d5908722-af08-4f95-8d85-e4ad444092b4" containerID="68e372784c1646fa51cb954a0833cc2ead868db5d6b5c408413f9d83aca65eb6" exitCode=0
Sep 18 18:54:32.649038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:32.648605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp" event={"ID":"d5908722-af08-4f95-8d85-e4ad444092b4","Type":"ContainerDied","Data":"68e372784c1646fa51cb954a0833cc2ead868db5d6b5c408413f9d83aca65eb6"}
Sep 18 18:54:33.795006 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:33.794976    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp"
Sep 18 18:54:33.905994 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:33.905960    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5908722-af08-4f95-8d85-e4ad444092b4-maas-api-service-ca\") pod \"d5908722-af08-4f95-8d85-e4ad444092b4\" (UID: \"d5908722-af08-4f95-8d85-e4ad444092b4\") "
Sep 18 18:54:33.906352 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:33.906330    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5908722-af08-4f95-8d85-e4ad444092b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5908722-af08-4f95-8d85-e4ad444092b4" (UID: "d5908722-af08-4f95-8d85-e4ad444092b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:54:34.007680 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:34.007595    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5908722-af08-4f95-8d85-e4ad444092b4-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:34.665117 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:34.665089    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp"
Sep 18 18:54:34.665297 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:34.665095    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp" event={"ID":"d5908722-af08-4f95-8d85-e4ad444092b4","Type":"ContainerDied","Data":"4af17a1ac48769c1f85b9a55a7868b5c697c2fa021d630baf3626b24fe95d6ac"}
Sep 18 18:54:34.665297 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:34.665254    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4af17a1ac48769c1f85b9a55a7868b5c697c2fa021d630baf3626b24fe95d6ac"
Sep 18 18:54:34.666984 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:34.666961    2558 generic.go:356] "Generic (PLEG): container finished" podID="46118a5a-9491-4cf0-a2a2-84e76137516a" containerID="b1115ae7fe3d235489900c62f2268c07a1776195f32693d8768dbf83b470c564" exitCode=0
Sep 18 18:54:34.667096 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:34.667013    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx" event={"ID":"46118a5a-9491-4cf0-a2a2-84e76137516a","Type":"ContainerDied","Data":"b1115ae7fe3d235489900c62f2268c07a1776195f32693d8768dbf83b470c564"}
Sep 18 18:54:34.667532 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:34.667515    2558 scope.go:117] "RemoveContainer" containerID="b1115ae7fe3d235489900c62f2268c07a1776195f32693d8768dbf83b470c564"
Sep 18 18:54:34.668657 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:34.668628    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:54:34.668729 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:34.668697    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:54:34.668772 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:34.668732    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:54:35.678447 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:35.678412    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx" event={"ID":"46118a5a-9491-4cf0-a2a2-84e76137516a","Type":"ContainerStarted","Data":"1f8ed7cd7a0e19e6d8014a885fb3d1500b8b7cd26f9647b52207e419306084d4"}
Sep 18 18:54:36.074825 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:36.074724    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:36.076163 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:36.076135    2558 patch_prober.go:28] interesting pod/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 18 18:54:36.076268 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:36.076212    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx" podUID="46118a5a-9491-4cf0-a2a2-84e76137516a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 18 18:54:37.073998 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:37.073958    2558 patch_prober.go:28] interesting pod/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 18 18:54:37.074449 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:37.074028    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx" podUID="46118a5a-9491-4cf0-a2a2-84e76137516a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 18 18:54:38.074406 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:38.074360    2558 patch_prober.go:28] interesting pod/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 18 18:54:38.074961 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:38.074432    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx" podUID="46118a5a-9491-4cf0-a2a2-84e76137516a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 18 18:54:39.074210 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:39.074161    2558 patch_prober.go:28] interesting pod/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 18 18:54:39.074429 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:39.074265    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx" podUID="46118a5a-9491-4cf0-a2a2-84e76137516a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 18 18:54:40.074046 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:40.074009    2558 patch_prober.go:28] interesting pod/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 18 18:54:40.074283 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:40.074076    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx" podUID="46118a5a-9491-4cf0-a2a2-84e76137516a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 18 18:54:40.468873 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:40.468831    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:41.079018 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:41.078987    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:41.079900 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:41.079877    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:41.463177 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:41.463134    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp"]
Sep 18 18:54:41.467585 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:41.467548    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e8c645ee-2h5hp"]
Sep 18 18:54:41.590371 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:41.590336    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"]
Sep 18 18:54:41.718034 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:41.717951    2558 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx" secret="" err="secret \"e2e-migrate-e8c645ee-gw-openshift-default-dockercfg-bzv59\" not found"
Sep 18 18:54:42.368856 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.368823    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"]
Sep 18 18:54:42.369356 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.369340    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5908722-af08-4f95-8d85-e4ad444092b4" containerName="revoke-keys"
Sep 18 18:54:42.369449 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.369359    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5908722-af08-4f95-8d85-e4ad444092b4" containerName="revoke-keys"
Sep 18 18:54:42.369449 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.369443    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5908722-af08-4f95-8d85-e4ad444092b4" containerName="revoke-keys"
Sep 18 18:54:42.372892 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.372871    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.375542 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.375525    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b0eada48-openshift-default-dockercfg-pfmk8\""
Sep 18 18:54:42.388559 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.388531    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"]
Sep 18 18:54:42.397043 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.397018    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-istio-data\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.397157 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.397059    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be22bf0-fa05-4407-a677-71dde44bc681-istio-token\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.397157 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.397146    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-workload-socket\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.397276 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.397180    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be22bf0-fa05-4407-a677-71dde44bc681-istiod-ca-cert\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.397276 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.397246    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-workload-certs\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.397372 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.397271    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh8wr\" (UniqueName: \"kubernetes.io/projected/6be22bf0-fa05-4407-a677-71dde44bc681-kube-api-access-nh8wr\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.397427 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.397392    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-istio-envoy\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.397487 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.397430    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-credential-socket\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.397487 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.397472    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be22bf0-fa05-4407-a677-71dde44bc681-istio-podinfo\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.498325 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.498290    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-istio-data\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.498325 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.498326    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be22bf0-fa05-4407-a677-71dde44bc681-istio-token\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.498582 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.498373    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-workload-socket\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.498582 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.498396    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be22bf0-fa05-4407-a677-71dde44bc681-istiod-ca-cert\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.498582 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.498422    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-workload-certs\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.498582 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.498443    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh8wr\" (UniqueName: \"kubernetes.io/projected/6be22bf0-fa05-4407-a677-71dde44bc681-kube-api-access-nh8wr\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.498582 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.498524    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-istio-envoy\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.498861 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.498591    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-credential-socket\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.498861 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.498647    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be22bf0-fa05-4407-a677-71dde44bc681-istio-podinfo\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.498861 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.498725    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-istio-data\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.498861 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.498776    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-workload-socket\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.499074 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.498914    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-workload-certs\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.499074 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.499037    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-credential-socket\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.499427 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.499403    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be22bf0-fa05-4407-a677-71dde44bc681-istiod-ca-cert\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.501150 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.501118    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-istio-envoy\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.501318 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.501290    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be22bf0-fa05-4407-a677-71dde44bc681-istio-podinfo\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.506901 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.506876    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be22bf0-fa05-4407-a677-71dde44bc681-istio-token\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.507176 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.507157    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh8wr\" (UniqueName: \"kubernetes.io/projected/6be22bf0-fa05-4407-a677-71dde44bc681-kube-api-access-nh8wr\") pod \"e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") " pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.577114 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.577081    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5908722-af08-4f95-8d85-e4ad444092b4" path="/var/lib/kubelet/pods/d5908722-af08-4f95-8d85-e4ad444092b4/volumes"
Sep 18 18:54:42.684015 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.683923    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:42.724764 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.724722    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx" podUID="46118a5a-9491-4cf0-a2a2-84e76137516a" containerName="istio-proxy" containerID="cri-o://1f8ed7cd7a0e19e6d8014a885fb3d1500b8b7cd26f9647b52207e419306084d4" gracePeriod=30
Sep 18 18:54:42.966057 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.966033    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"]
Sep 18 18:54:42.967598 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:54:42.967572    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be22bf0_fa05_4407_a677_71dde44bc681.slice/crio-8e7a0927df2cefb4e50a97cb73c93e3b8260ff48640d83326d9612f2e669678d WatchSource:0}: Error finding container 8e7a0927df2cefb4e50a97cb73c93e3b8260ff48640d83326d9612f2e669678d: Status 404 returned error can't find the container with id 8e7a0927df2cefb4e50a97cb73c93e3b8260ff48640d83326d9612f2e669678d
Sep 18 18:54:42.970099 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.970065    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:54:42.970221 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.970137    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:54:42.970221 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:42.970166    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:54:43.733264 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:43.733226    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk" event={"ID":"6be22bf0-fa05-4407-a677-71dde44bc681","Type":"ContainerStarted","Data":"6f739acb1ac56c8a4298eeaaeb6a43b596892d0d95483a652f7646ca350bdb30"}
Sep 18 18:54:43.733738 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:43.733292    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk" event={"ID":"6be22bf0-fa05-4407-a677-71dde44bc681","Type":"ContainerStarted","Data":"8e7a0927df2cefb4e50a97cb73c93e3b8260ff48640d83326d9612f2e669678d"}
Sep 18 18:54:43.754753 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:43.754697    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk" podStartSLOduration=1.7546723709999998 podStartE2EDuration="1.754672371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:54:43.751502469 +0000 UTC m=+943.188081310" watchObservedRunningTime="2026-09-18 18:54:43.754672371 +0000 UTC m=+943.191251212"
Sep 18 18:54:44.684785 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:44.684746    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:44.690202 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:44.690165    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:44.739126 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:44.739093    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:44.740104 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:44.740082    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:54:47.991471 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:47.991414    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:48.057828 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.057793    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46118a5a-9491-4cf0-a2a2-84e76137516a-istiod-ca-cert\") pod \"46118a5a-9491-4cf0-a2a2-84e76137516a\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") "
Sep 18 18:54:48.057828 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.057834    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-envoy\") pod \"46118a5a-9491-4cf0-a2a2-84e76137516a\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") "
Sep 18 18:54:48.058057 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.057859    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-workload-certs\") pod \"46118a5a-9491-4cf0-a2a2-84e76137516a\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") "
Sep 18 18:54:48.058057 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.057922    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-credential-socket\") pod \"46118a5a-9491-4cf0-a2a2-84e76137516a\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") "
Sep 18 18:54:48.058057 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.057938    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-workload-socket\") pod \"46118a5a-9491-4cf0-a2a2-84e76137516a\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") "
Sep 18 18:54:48.058057 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.057977    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-token\") pod \"46118a5a-9491-4cf0-a2a2-84e76137516a\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") "
Sep 18 18:54:48.058057 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.058004    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-data\") pod \"46118a5a-9491-4cf0-a2a2-84e76137516a\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") "
Sep 18 18:54:48.058057 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.058029    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-podinfo\") pod \"46118a5a-9491-4cf0-a2a2-84e76137516a\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") "
Sep 18 18:54:48.058374 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.058097    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmnrx\" (UniqueName: \"kubernetes.io/projected/46118a5a-9491-4cf0-a2a2-84e76137516a-kube-api-access-qmnrx\") pod \"46118a5a-9491-4cf0-a2a2-84e76137516a\" (UID: \"46118a5a-9491-4cf0-a2a2-84e76137516a\") "
Sep 18 18:54:48.058374 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.058244    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "46118a5a-9491-4cf0-a2a2-84e76137516a" (UID: "46118a5a-9491-4cf0-a2a2-84e76137516a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:54:48.058374 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.058276    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "46118a5a-9491-4cf0-a2a2-84e76137516a" (UID: "46118a5a-9491-4cf0-a2a2-84e76137516a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:54:48.058374 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.058309    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-data" (OuterVolumeSpecName: "istio-data") pod "46118a5a-9491-4cf0-a2a2-84e76137516a" (UID: "46118a5a-9491-4cf0-a2a2-84e76137516a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:54:48.058374 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.058283    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "46118a5a-9491-4cf0-a2a2-84e76137516a" (UID: "46118a5a-9491-4cf0-a2a2-84e76137516a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:54:48.058612 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.058462    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46118a5a-9491-4cf0-a2a2-84e76137516a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "46118a5a-9491-4cf0-a2a2-84e76137516a" (UID: "46118a5a-9491-4cf0-a2a2-84e76137516a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:54:48.058612 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.058588    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46118a5a-9491-4cf0-a2a2-84e76137516a-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:48.058612 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.058610    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:48.058742 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.058623    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:48.058742 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.058639    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:48.058742 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.058652    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:48.060205 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.060143    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-token" (OuterVolumeSpecName: "istio-token") pod "46118a5a-9491-4cf0-a2a2-84e76137516a" (UID: "46118a5a-9491-4cf0-a2a2-84e76137516a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:54:48.060205 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.060170    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "46118a5a-9491-4cf0-a2a2-84e76137516a" (UID: "46118a5a-9491-4cf0-a2a2-84e76137516a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:54:48.060522 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.060497    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "46118a5a-9491-4cf0-a2a2-84e76137516a" (UID: "46118a5a-9491-4cf0-a2a2-84e76137516a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:54:48.060522 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.060511    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46118a5a-9491-4cf0-a2a2-84e76137516a-kube-api-access-qmnrx" (OuterVolumeSpecName: "kube-api-access-qmnrx") pod "46118a5a-9491-4cf0-a2a2-84e76137516a" (UID: "46118a5a-9491-4cf0-a2a2-84e76137516a"). InnerVolumeSpecName "kube-api-access-qmnrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:54:48.159328 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.159286    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmnrx\" (UniqueName: \"kubernetes.io/projected/46118a5a-9491-4cf0-a2a2-84e76137516a-kube-api-access-qmnrx\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:48.159328 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.159321    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:48.159328 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.159335    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:48.159551 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.159345    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46118a5a-9491-4cf0-a2a2-84e76137516a-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:54:48.766896 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.766859    2558 generic.go:356] "Generic (PLEG): container finished" podID="46118a5a-9491-4cf0-a2a2-84e76137516a" containerID="1f8ed7cd7a0e19e6d8014a885fb3d1500b8b7cd26f9647b52207e419306084d4" exitCode=0
Sep 18 18:54:48.767061 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.766906    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"
Sep 18 18:54:48.767061 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.766921    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx" event={"ID":"46118a5a-9491-4cf0-a2a2-84e76137516a","Type":"ContainerDied","Data":"1f8ed7cd7a0e19e6d8014a885fb3d1500b8b7cd26f9647b52207e419306084d4"}
Sep 18 18:54:48.767061 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.766969    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx" event={"ID":"46118a5a-9491-4cf0-a2a2-84e76137516a","Type":"ContainerDied","Data":"47458d422b3218094c0e4506c300d9d02c6b997ae25e651a8e18076c73981fca"}
Sep 18 18:54:48.767061 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.766993    2558 scope.go:117] "RemoveContainer" containerID="1f8ed7cd7a0e19e6d8014a885fb3d1500b8b7cd26f9647b52207e419306084d4"
Sep 18 18:54:48.776497 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.776479    2558 scope.go:117] "RemoveContainer" containerID="b1115ae7fe3d235489900c62f2268c07a1776195f32693d8768dbf83b470c564"
Sep 18 18:54:48.784560 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.784538    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"]
Sep 18 18:54:48.785491 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.785471    2558 scope.go:117] "RemoveContainer" containerID="1f8ed7cd7a0e19e6d8014a885fb3d1500b8b7cd26f9647b52207e419306084d4"
Sep 18 18:54:48.785781 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:54:48.785762    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8ed7cd7a0e19e6d8014a885fb3d1500b8b7cd26f9647b52207e419306084d4\": container with ID starting with 1f8ed7cd7a0e19e6d8014a885fb3d1500b8b7cd26f9647b52207e419306084d4 not found: ID does not exist" containerID="1f8ed7cd7a0e19e6d8014a885fb3d1500b8b7cd26f9647b52207e419306084d4"
Sep 18 18:54:48.785837 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.785791    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8ed7cd7a0e19e6d8014a885fb3d1500b8b7cd26f9647b52207e419306084d4"} err="failed to get container status \"1f8ed7cd7a0e19e6d8014a885fb3d1500b8b7cd26f9647b52207e419306084d4\": rpc error: code = NotFound desc = could not find container \"1f8ed7cd7a0e19e6d8014a885fb3d1500b8b7cd26f9647b52207e419306084d4\": container with ID starting with 1f8ed7cd7a0e19e6d8014a885fb3d1500b8b7cd26f9647b52207e419306084d4 not found: ID does not exist"
Sep 18 18:54:48.785837 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.785812    2558 scope.go:117] "RemoveContainer" containerID="b1115ae7fe3d235489900c62f2268c07a1776195f32693d8768dbf83b470c564"
Sep 18 18:54:48.786051 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:54:48.786032    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1115ae7fe3d235489900c62f2268c07a1776195f32693d8768dbf83b470c564\": container with ID starting with b1115ae7fe3d235489900c62f2268c07a1776195f32693d8768dbf83b470c564 not found: ID does not exist" containerID="b1115ae7fe3d235489900c62f2268c07a1776195f32693d8768dbf83b470c564"
Sep 18 18:54:48.786098 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.786057    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1115ae7fe3d235489900c62f2268c07a1776195f32693d8768dbf83b470c564"} err="failed to get container status \"b1115ae7fe3d235489900c62f2268c07a1776195f32693d8768dbf83b470c564\": rpc error: code = NotFound desc = could not find container \"b1115ae7fe3d235489900c62f2268c07a1776195f32693d8768dbf83b470c564\": container with ID starting with b1115ae7fe3d235489900c62f2268c07a1776195f32693d8768dbf83b470c564 not found: ID does not exist"
Sep 18 18:54:48.795982 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:48.795951    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e8c645ee-gw-openshift-default-5b678599d8-f5rtx"]
Sep 18 18:54:50.578717 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:50.578675    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46118a5a-9491-4cf0-a2a2-84e76137516a" path="/var/lib/kubelet/pods/46118a5a-9491-4cf0-a2a2-84e76137516a/volumes"
Sep 18 18:54:51.130174 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.130137    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"]
Sep 18 18:54:51.130612 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.130598    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46118a5a-9491-4cf0-a2a2-84e76137516a" containerName="istio-proxy"
Sep 18 18:54:51.130655 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.130613    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="46118a5a-9491-4cf0-a2a2-84e76137516a" containerName="istio-proxy"
Sep 18 18:54:51.130717 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.130708    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="46118a5a-9491-4cf0-a2a2-84e76137516a" containerName="istio-proxy"
Sep 18 18:54:51.130752 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.130721    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="46118a5a-9491-4cf0-a2a2-84e76137516a" containerName="istio-proxy"
Sep 18 18:54:51.133959 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.133943    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.137144 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.137121    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bcb770\"/\"default-dockercfg-d5dsj\""
Sep 18 18:54:51.137144 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.137134    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bcb770\"/\"openshift-service-ca.crt\""
Sep 18 18:54:51.137327 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.137157    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bcb770\"/\"test-model-bcb770-kserve-self-signed-certs\""
Sep 18 18:54:51.137327 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.137163    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bcb770\"/\"kube-root-ca.crt\""
Sep 18 18:54:51.143538 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.143518    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"]
Sep 18 18:54:51.191043 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.191010    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-dshm\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.191230 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.191067    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-model-cache\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.191230 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.191098    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-home\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.191230 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.191138    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3dd0b58-1324-4295-9425-8990c8959583-tls-certs\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.191344 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.191267    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt92d\" (UniqueName: \"kubernetes.io/projected/f3dd0b58-1324-4295-9425-8990c8959583-kube-api-access-qt92d\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.191344 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.191288    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-tmp-dir\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.292132 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.292099    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-dshm\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.292332 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.292149    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-model-cache\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.292332 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.292264    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-home\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.292449 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.292345    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3dd0b58-1324-4295-9425-8990c8959583-tls-certs\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.292518 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.292456    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt92d\" (UniqueName: \"kubernetes.io/projected/f3dd0b58-1324-4295-9425-8990c8959583-kube-api-access-qt92d\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.292518 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.292490    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-tmp-dir\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.292631 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.292518    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-model-cache\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.292682 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.292645    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-home\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.292863 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.292837    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-tmp-dir\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.294471 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.294451    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-dshm\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.294851 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.294832    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3dd0b58-1324-4295-9425-8990c8959583-tls-certs\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.299236 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.299217    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt92d\" (UniqueName: \"kubernetes.io/projected/f3dd0b58-1324-4295-9425-8990c8959583-kube-api-access-qt92d\") pod \"test-model-bcb770-kserve-564498dc-fb586\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") " pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.444459 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.444378    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:54:51.718540 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.718510    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"]
Sep 18 18:54:51.720454 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:54:51.720426    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3dd0b58_1324_4295_9425_8990c8959583.slice/crio-8443b37606377f3b439ef964b1ba7710883b2c20a30c45ced6998a4ca1797085 WatchSource:0}: Error finding container 8443b37606377f3b439ef964b1ba7710883b2c20a30c45ced6998a4ca1797085: Status 404 returned error can't find the container with id 8443b37606377f3b439ef964b1ba7710883b2c20a30c45ced6998a4ca1797085
Sep 18 18:54:51.789930 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:51.789894    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586" event={"ID":"f3dd0b58-1324-4295-9425-8990c8959583","Type":"ContainerStarted","Data":"8443b37606377f3b439ef964b1ba7710883b2c20a30c45ced6998a4ca1797085"}
Sep 18 18:54:52.798208 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:52.798156    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586" event={"ID":"f3dd0b58-1324-4295-9425-8990c8959583","Type":"ContainerStarted","Data":"86cb1e5593c1bb158cb3dee17d25fb62161b0caba642a6feb93e477e96572cc3"}
Sep 18 18:54:52.816215 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:54:52.816155    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586" podStartSLOduration=1.81614087 podStartE2EDuration="1.81614087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:54:52.814491925 +0000 UTC m=+952.251070761" watchObservedRunningTime="2026-09-18 18:54:52.81614087 +0000 UTC m=+952.252719713"
Sep 18 18:55:01.445220 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:01.445168    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:55:01.445823 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:01.445245    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:55:01.484877 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:01.484850    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:55:01.883922 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:01.883880    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:55:07.353420 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.353386    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"]
Sep 18 18:55:07.353840 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.353827    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46118a5a-9491-4cf0-a2a2-84e76137516a" containerName="istio-proxy"
Sep 18 18:55:07.353840 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.353841    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="46118a5a-9491-4cf0-a2a2-84e76137516a" containerName="istio-proxy"
Sep 18 18:55:07.358409 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.358389    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.360875 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.360855    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9d9f36\"/\"premium-w3-9d9f36-kserve-self-signed-certs\""
Sep 18 18:55:07.365733 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.365705    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62c2704-fde4-459c-8299-f32c41f5a4ac-tls-certs\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.365856 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.365798    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-home\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.365927 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.365854    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-dshm\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.365927 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.365897    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpr6v\" (UniqueName: \"kubernetes.io/projected/a62c2704-fde4-459c-8299-f32c41f5a4ac-kube-api-access-zpr6v\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.366041 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.366023    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-model-cache\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.366095 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.366064    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-tmp-dir\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.366476 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.366455    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"]
Sep 18 18:55:07.466973 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.466938    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-tmp-dir\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.466973 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.466977    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62c2704-fde4-459c-8299-f32c41f5a4ac-tls-certs\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.467252 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.467003    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-home\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.467252 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.467040    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-dshm\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.467252 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.467068    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpr6v\" (UniqueName: \"kubernetes.io/projected/a62c2704-fde4-459c-8299-f32c41f5a4ac-kube-api-access-zpr6v\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.467252 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.467201    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-model-cache\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.467493 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.467355    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-home\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.467493 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.467439    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-tmp-dir\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.467731 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.467714    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-model-cache\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.469569 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.469540    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-dshm\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.469692 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.469653    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62c2704-fde4-459c-8299-f32c41f5a4ac-tls-certs\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.475316 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.475293    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpr6v\" (UniqueName: \"kubernetes.io/projected/a62c2704-fde4-459c-8299-f32c41f5a4ac-kube-api-access-zpr6v\") pod \"premium-w3-9d9f36-kserve-845474c47b-c9npl\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") " pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.671249 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.671216    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:07.904690 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.904602    2558 generic.go:356] "Generic (PLEG): container finished" podID="6be22bf0-fa05-4407-a677-71dde44bc681" containerID="6f739acb1ac56c8a4298eeaaeb6a43b596892d0d95483a652f7646ca350bdb30" exitCode=0
Sep 18 18:55:07.904690 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.904673    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk" event={"ID":"6be22bf0-fa05-4407-a677-71dde44bc681","Type":"ContainerDied","Data":"6f739acb1ac56c8a4298eeaaeb6a43b596892d0d95483a652f7646ca350bdb30"}
Sep 18 18:55:07.905129 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.905115    2558 scope.go:117] "RemoveContainer" containerID="6f739acb1ac56c8a4298eeaaeb6a43b596892d0d95483a652f7646ca350bdb30"
Sep 18 18:55:07.906298 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.906268    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:55:07.906398 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.906337    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:55:07.906398 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.906366    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:55:07.952089 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:07.952062    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"]
Sep 18 18:55:07.954407 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:55:07.954359    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda62c2704_fde4_459c_8299_f32c41f5a4ac.slice/crio-bea74ca851d2785571dd7889cd67152f1c97daaf2552dccd1cca003ccf08dd6e WatchSource:0}: Error finding container bea74ca851d2785571dd7889cd67152f1c97daaf2552dccd1cca003ccf08dd6e: Status 404 returned error can't find the container with id bea74ca851d2785571dd7889cd67152f1c97daaf2552dccd1cca003ccf08dd6e
Sep 18 18:55:08.913249 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:08.913215    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk" event={"ID":"6be22bf0-fa05-4407-a677-71dde44bc681","Type":"ContainerStarted","Data":"e0c57aa6acee29c9d4249f110b0845537fb3e3abb0060bb32ee288b032df26cd"}
Sep 18 18:55:08.914969 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:08.914944    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl" event={"ID":"a62c2704-fde4-459c-8299-f32c41f5a4ac","Type":"ContainerStarted","Data":"14801e399ad02c7bd2de0d16ae9edea01b2bb5e1914b4c470151691df25deb03"}
Sep 18 18:55:08.915130 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:08.914978    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl" event={"ID":"a62c2704-fde4-459c-8299-f32c41f5a4ac","Type":"ContainerStarted","Data":"bea74ca851d2785571dd7889cd67152f1c97daaf2552dccd1cca003ccf08dd6e"}
Sep 18 18:55:08.963018 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:08.962939    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl" podStartSLOduration=1.9629150370000001 podStartE2EDuration="1.962915037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:55:08.958547216 +0000 UTC m=+968.395126061" watchObservedRunningTime="2026-09-18 18:55:08.962915037 +0000 UTC m=+968.399493880"
Sep 18 18:55:09.684591 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:09.684550    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:55:09.684841 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:09.684817    2558 patch_prober.go:28] interesting pod/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 18 18:55:09.684890 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:09.684869    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk" podUID="6be22bf0-fa05-4407-a677-71dde44bc681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 18 18:55:10.684562 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:10.684527    2558 patch_prober.go:28] interesting pod/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 18 18:55:10.684956 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:10.684584    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk" podUID="6be22bf0-fa05-4407-a677-71dde44bc681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 18 18:55:11.684767 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:11.684729    2558 patch_prober.go:28] interesting pod/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 18 18:55:11.685279 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:11.684785    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk" podUID="6be22bf0-fa05-4407-a677-71dde44bc681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 18 18:55:12.685017 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:12.684975    2558 patch_prober.go:28] interesting pod/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 18 18:55:12.685467 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:12.685039    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk" podUID="6be22bf0-fa05-4407-a677-71dde44bc681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 18 18:55:13.685303 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:13.685265    2558 patch_prober.go:28] interesting pod/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 18 18:55:13.685707 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:13.685322    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk" podUID="6be22bf0-fa05-4407-a677-71dde44bc681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 18 18:55:14.689227 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:14.689179    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:55:14.689644 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:14.689627    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:55:14.690401 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:14.690381    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:55:15.996041 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:15.995960    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm"]
Sep 18 18:55:15.999945 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:15.999924    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm"
Sep 18 18:55:16.004829 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:16.004806    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 18:55:16.004917 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:16.004870    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 18:55:16.014923 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:16.014900    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm"]
Sep 18 18:55:16.054428 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:16.054395    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6cba06b-4426-40ba-820e-9d8703bea665-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm\" (UID: \"e6cba06b-4426-40ba-820e-9d8703bea665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm"
Sep 18 18:55:16.156008 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:16.155974    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6cba06b-4426-40ba-820e-9d8703bea665-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm\" (UID: \"e6cba06b-4426-40ba-820e-9d8703bea665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm"
Sep 18 18:55:16.156595 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:16.156577    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6cba06b-4426-40ba-820e-9d8703bea665-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm\" (UID: \"e6cba06b-4426-40ba-820e-9d8703bea665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm"
Sep 18 18:55:16.311049 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:16.310954    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm"
Sep 18 18:55:16.584446 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:16.584420    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm"]
Sep 18 18:55:16.586083 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:55:16.586061    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6cba06b_4426_40ba_820e_9d8703bea665.slice/crio-1764c8136a8329a995c85f7ca269b67178c194f59fdd56be0b74b4576b053c1b WatchSource:0}: Error finding container 1764c8136a8329a995c85f7ca269b67178c194f59fdd56be0b74b4576b053c1b: Status 404 returned error can't find the container with id 1764c8136a8329a995c85f7ca269b67178c194f59fdd56be0b74b4576b053c1b
Sep 18 18:55:16.966718 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:16.966676    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm" event={"ID":"e6cba06b-4426-40ba-820e-9d8703bea665","Type":"ContainerStarted","Data":"cecb78b391b0feaaf4f5c6a1c4fb162c766d2a80298342a15c8ac8ffccfcbf76"}
Sep 18 18:55:16.966885 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:16.966741    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm" event={"ID":"e6cba06b-4426-40ba-820e-9d8703bea665","Type":"ContainerStarted","Data":"1764c8136a8329a995c85f7ca269b67178c194f59fdd56be0b74b4576b053c1b"}
Sep 18 18:55:16.982312 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:16.982259    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm" podStartSLOduration=1.982245121 podStartE2EDuration="1.982245121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:55:16.980836099 +0000 UTC m=+976.417414940" watchObservedRunningTime="2026-09-18 18:55:16.982245121 +0000 UTC m=+976.418823964"
Sep 18 18:55:17.672253 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:17.672147    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:17.672253 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:17.672212    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:17.685756 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:17.685729    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:17.978641 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:17.978530    2558 generic.go:356] "Generic (PLEG): container finished" podID="e6cba06b-4426-40ba-820e-9d8703bea665" containerID="cecb78b391b0feaaf4f5c6a1c4fb162c766d2a80298342a15c8ac8ffccfcbf76" exitCode=0
Sep 18 18:55:17.980425 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:17.980391    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm" event={"ID":"e6cba06b-4426-40ba-820e-9d8703bea665","Type":"ContainerDied","Data":"cecb78b391b0feaaf4f5c6a1c4fb162c766d2a80298342a15c8ac8ffccfcbf76"}
Sep 18 18:55:17.997494 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:17.997464    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 18:55:19.128843 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:19.128815    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm"
Sep 18 18:55:19.189884 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:19.189851    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6cba06b-4426-40ba-820e-9d8703bea665-maas-api-service-ca\") pod \"e6cba06b-4426-40ba-820e-9d8703bea665\" (UID: \"e6cba06b-4426-40ba-820e-9d8703bea665\") "
Sep 18 18:55:19.190219 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:19.190172    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6cba06b-4426-40ba-820e-9d8703bea665-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e6cba06b-4426-40ba-820e-9d8703bea665" (UID: "e6cba06b-4426-40ba-820e-9d8703bea665"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:55:19.291705 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:19.291618    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6cba06b-4426-40ba-820e-9d8703bea665-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:55:19.993586 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:19.993536    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm" event={"ID":"e6cba06b-4426-40ba-820e-9d8703bea665","Type":"ContainerDied","Data":"1764c8136a8329a995c85f7ca269b67178c194f59fdd56be0b74b4576b053c1b"}
Sep 18 18:55:19.993806 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:19.993604    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1764c8136a8329a995c85f7ca269b67178c194f59fdd56be0b74b4576b053c1b"
Sep 18 18:55:19.993806 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:19.993548    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm"
Sep 18 18:55:27.996914 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:27.996881    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm"]
Sep 18 18:55:28.001297 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:28.001266    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0eada48-pmqpm"]
Sep 18 18:55:28.577041 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:28.577009    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6cba06b-4426-40ba-820e-9d8703bea665" path="/var/lib/kubelet/pods/e6cba06b-4426-40ba-820e-9d8703bea665/volumes"
Sep 18 18:55:29.177116 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:29.177086    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"]
Sep 18 18:55:29.177554 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:29.177315    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk" podUID="6be22bf0-fa05-4407-a677-71dde44bc681" containerName="istio-proxy" containerID="cri-o://e0c57aa6acee29c9d4249f110b0845537fb3e3abb0060bb32ee288b032df26cd" gracePeriod=30
Sep 18 18:55:29.692345 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:29.692310    2558 patch_prober.go:28] interesting pod/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 18:55:29.692525 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:29.692367    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk" podUID="6be22bf0-fa05-4407-a677-71dde44bc681" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 18:55:34.456250 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.456176    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:55:34.467042 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467015    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-workload-socket\") pod \"6be22bf0-fa05-4407-a677-71dde44bc681\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") "
Sep 18 18:55:34.467168 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467046    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be22bf0-fa05-4407-a677-71dde44bc681-istiod-ca-cert\") pod \"6be22bf0-fa05-4407-a677-71dde44bc681\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") "
Sep 18 18:55:34.467168 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467076    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-istio-envoy\") pod \"6be22bf0-fa05-4407-a677-71dde44bc681\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") "
Sep 18 18:55:34.467168 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467119    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be22bf0-fa05-4407-a677-71dde44bc681-istio-podinfo\") pod \"6be22bf0-fa05-4407-a677-71dde44bc681\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") "
Sep 18 18:55:34.467168 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467138    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-workload-certs\") pod \"6be22bf0-fa05-4407-a677-71dde44bc681\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") "
Sep 18 18:55:34.467413 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467175    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-istio-data\") pod \"6be22bf0-fa05-4407-a677-71dde44bc681\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") "
Sep 18 18:55:34.467413 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467211    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-credential-socket\") pod \"6be22bf0-fa05-4407-a677-71dde44bc681\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") "
Sep 18 18:55:34.467413 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467232    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh8wr\" (UniqueName: \"kubernetes.io/projected/6be22bf0-fa05-4407-a677-71dde44bc681-kube-api-access-nh8wr\") pod \"6be22bf0-fa05-4407-a677-71dde44bc681\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") "
Sep 18 18:55:34.467413 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467290    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be22bf0-fa05-4407-a677-71dde44bc681-istio-token\") pod \"6be22bf0-fa05-4407-a677-71dde44bc681\" (UID: \"6be22bf0-fa05-4407-a677-71dde44bc681\") "
Sep 18 18:55:34.467413 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467347    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6be22bf0-fa05-4407-a677-71dde44bc681" (UID: "6be22bf0-fa05-4407-a677-71dde44bc681"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:55:34.467669 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467468    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be22bf0-fa05-4407-a677-71dde44bc681-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6be22bf0-fa05-4407-a677-71dde44bc681" (UID: "6be22bf0-fa05-4407-a677-71dde44bc681"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:55:34.467669 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467574    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6be22bf0-fa05-4407-a677-71dde44bc681" (UID: "6be22bf0-fa05-4407-a677-71dde44bc681"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:55:34.467779 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467748    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:55:34.467779 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467774    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be22bf0-fa05-4407-a677-71dde44bc681-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:55:34.467878 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467794    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:55:34.467878 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467738    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6be22bf0-fa05-4407-a677-71dde44bc681" (UID: "6be22bf0-fa05-4407-a677-71dde44bc681"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:55:34.467878 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.467751    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-istio-data" (OuterVolumeSpecName: "istio-data") pod "6be22bf0-fa05-4407-a677-71dde44bc681" (UID: "6be22bf0-fa05-4407-a677-71dde44bc681"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:55:34.469433 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.469406    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6be22bf0-fa05-4407-a677-71dde44bc681" (UID: "6be22bf0-fa05-4407-a677-71dde44bc681"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:55:34.469616 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.469591    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6be22bf0-fa05-4407-a677-71dde44bc681-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6be22bf0-fa05-4407-a677-71dde44bc681" (UID: "6be22bf0-fa05-4407-a677-71dde44bc681"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:55:34.469616 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.469601    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be22bf0-fa05-4407-a677-71dde44bc681-kube-api-access-nh8wr" (OuterVolumeSpecName: "kube-api-access-nh8wr") pod "6be22bf0-fa05-4407-a677-71dde44bc681" (UID: "6be22bf0-fa05-4407-a677-71dde44bc681"). InnerVolumeSpecName "kube-api-access-nh8wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:55:34.469745 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.469624    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be22bf0-fa05-4407-a677-71dde44bc681-istio-token" (OuterVolumeSpecName: "istio-token") pod "6be22bf0-fa05-4407-a677-71dde44bc681" (UID: "6be22bf0-fa05-4407-a677-71dde44bc681"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:55:34.569217 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.569164    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:55:34.569394 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.569237    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be22bf0-fa05-4407-a677-71dde44bc681-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:55:34.569394 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.569281    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:55:34.569394 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.569332    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be22bf0-fa05-4407-a677-71dde44bc681-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:55:34.569531 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.569393    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh8wr\" (UniqueName: \"kubernetes.io/projected/6be22bf0-fa05-4407-a677-71dde44bc681-kube-api-access-nh8wr\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:55:34.569531 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:34.569522    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be22bf0-fa05-4407-a677-71dde44bc681-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:55:35.089297 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:35.089260    2558 generic.go:356] "Generic (PLEG): container finished" podID="6be22bf0-fa05-4407-a677-71dde44bc681" containerID="e0c57aa6acee29c9d4249f110b0845537fb3e3abb0060bb32ee288b032df26cd" exitCode=0
Sep 18 18:55:35.089517 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:35.089306    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"
Sep 18 18:55:35.089517 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:35.089311    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk" event={"ID":"6be22bf0-fa05-4407-a677-71dde44bc681","Type":"ContainerDied","Data":"e0c57aa6acee29c9d4249f110b0845537fb3e3abb0060bb32ee288b032df26cd"}
Sep 18 18:55:35.089517 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:35.089368    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk" event={"ID":"6be22bf0-fa05-4407-a677-71dde44bc681","Type":"ContainerDied","Data":"8e7a0927df2cefb4e50a97cb73c93e3b8260ff48640d83326d9612f2e669678d"}
Sep 18 18:55:35.089517 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:35.089396    2558 scope.go:117] "RemoveContainer" containerID="e0c57aa6acee29c9d4249f110b0845537fb3e3abb0060bb32ee288b032df26cd"
Sep 18 18:55:35.099026 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:35.099007    2558 scope.go:117] "RemoveContainer" containerID="6f739acb1ac56c8a4298eeaaeb6a43b596892d0d95483a652f7646ca350bdb30"
Sep 18 18:55:35.108239 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:35.108214    2558 scope.go:117] "RemoveContainer" containerID="e0c57aa6acee29c9d4249f110b0845537fb3e3abb0060bb32ee288b032df26cd"
Sep 18 18:55:35.108505 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:55:35.108485    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c57aa6acee29c9d4249f110b0845537fb3e3abb0060bb32ee288b032df26cd\": container with ID starting with e0c57aa6acee29c9d4249f110b0845537fb3e3abb0060bb32ee288b032df26cd not found: ID does not exist" containerID="e0c57aa6acee29c9d4249f110b0845537fb3e3abb0060bb32ee288b032df26cd"
Sep 18 18:55:35.108564 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:35.108517    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c57aa6acee29c9d4249f110b0845537fb3e3abb0060bb32ee288b032df26cd"} err="failed to get container status \"e0c57aa6acee29c9d4249f110b0845537fb3e3abb0060bb32ee288b032df26cd\": rpc error: code = NotFound desc = could not find container \"e0c57aa6acee29c9d4249f110b0845537fb3e3abb0060bb32ee288b032df26cd\": container with ID starting with e0c57aa6acee29c9d4249f110b0845537fb3e3abb0060bb32ee288b032df26cd not found: ID does not exist"
Sep 18 18:55:35.108564 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:35.108538    2558 scope.go:117] "RemoveContainer" containerID="6f739acb1ac56c8a4298eeaaeb6a43b596892d0d95483a652f7646ca350bdb30"
Sep 18 18:55:35.108808 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:55:35.108791    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f739acb1ac56c8a4298eeaaeb6a43b596892d0d95483a652f7646ca350bdb30\": container with ID starting with 6f739acb1ac56c8a4298eeaaeb6a43b596892d0d95483a652f7646ca350bdb30 not found: ID does not exist" containerID="6f739acb1ac56c8a4298eeaaeb6a43b596892d0d95483a652f7646ca350bdb30"
Sep 18 18:55:35.108864 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:35.108813    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f739acb1ac56c8a4298eeaaeb6a43b596892d0d95483a652f7646ca350bdb30"} err="failed to get container status \"6f739acb1ac56c8a4298eeaaeb6a43b596892d0d95483a652f7646ca350bdb30\": rpc error: code = NotFound desc = could not find container \"6f739acb1ac56c8a4298eeaaeb6a43b596892d0d95483a652f7646ca350bdb30\": container with ID starting with 6f739acb1ac56c8a4298eeaaeb6a43b596892d0d95483a652f7646ca350bdb30 not found: ID does not exist"
Sep 18 18:55:35.109487 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:35.109469    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"]
Sep 18 18:55:35.113859 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:35.113836    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b0eada48-openshift-default-5d7bf7dd58-k6qgk"]
Sep 18 18:55:36.576598 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:36.576563    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6be22bf0-fa05-4407-a677-71dde44bc681" path="/var/lib/kubelet/pods/6be22bf0-fa05-4407-a677-71dde44bc681/volumes"
Sep 18 18:55:37.831925 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.831892    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"]
Sep 18 18:55:37.832347 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.832333    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be22bf0-fa05-4407-a677-71dde44bc681" containerName="istio-proxy"
Sep 18 18:55:37.832395 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.832348    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be22bf0-fa05-4407-a677-71dde44bc681" containerName="istio-proxy"
Sep 18 18:55:37.832395 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.832362    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6cba06b-4426-40ba-820e-9d8703bea665" containerName="revoke-keys"
Sep 18 18:55:37.832395 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.832367    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6cba06b-4426-40ba-820e-9d8703bea665" containerName="revoke-keys"
Sep 18 18:55:37.832395 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.832388    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be22bf0-fa05-4407-a677-71dde44bc681" containerName="istio-proxy"
Sep 18 18:55:37.832395 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.832395    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be22bf0-fa05-4407-a677-71dde44bc681" containerName="istio-proxy"
Sep 18 18:55:37.832550 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.832453    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be22bf0-fa05-4407-a677-71dde44bc681" containerName="istio-proxy"
Sep 18 18:55:37.832550 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.832459    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6cba06b-4426-40ba-820e-9d8703bea665" containerName="revoke-keys"
Sep 18 18:55:37.832550 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.832466    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be22bf0-fa05-4407-a677-71dde44bc681" containerName="istio-proxy"
Sep 18 18:55:37.837057 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.837042    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:37.839480 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.839456    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-cc289c\"/\"premium-w0-cc289c-kserve-self-signed-certs\""
Sep 18 18:55:37.846469 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.846437    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"]
Sep 18 18:55:37.901800 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.901767    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-model-cache\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:37.901971 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.901809    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-tls-certs\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:37.901971 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.901827    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55fbx\" (UniqueName: \"kubernetes.io/projected/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-kube-api-access-55fbx\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:37.901971 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.901945    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-home\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:37.901971 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.901968    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-dshm\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:37.902128 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:37.902001    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-tmp-dir\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:38.003231 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:38.003180    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-model-cache\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:38.003414 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:38.003258    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-tls-certs\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:38.003414 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:38.003287    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55fbx\" (UniqueName: \"kubernetes.io/projected/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-kube-api-access-55fbx\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:38.003414 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:38.003398    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-home\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:38.003584 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:38.003424    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-dshm\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:38.003921 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:38.003691    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-tmp-dir\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:38.003921 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:38.003693    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-home\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:38.003921 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:38.003610    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-model-cache\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:38.004114 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:38.003933    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-tmp-dir\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:38.006215 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:38.006146    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-tls-certs\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:38.006215 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:38.006156    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-dshm\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:38.013234 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:38.013162    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55fbx\" (UniqueName: \"kubernetes.io/projected/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-kube-api-access-55fbx\") pod \"premium-w0-cc289c-kserve-85d8f7fd8-bjr5x\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") " pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:38.147629 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:38.147595    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:38.425784 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:38.425755    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"]
Sep 18 18:55:38.427209 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:55:38.427157    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce76d297_042d_4fb1_8f79_dbe1ccfb546f.slice/crio-c3007013291cb86f197af3d800a2ee505f3d63698ba0b548a1fe8c2bcd7c8b52 WatchSource:0}: Error finding container c3007013291cb86f197af3d800a2ee505f3d63698ba0b548a1fe8c2bcd7c8b52: Status 404 returned error can't find the container with id c3007013291cb86f197af3d800a2ee505f3d63698ba0b548a1fe8c2bcd7c8b52
Sep 18 18:55:39.123877 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:39.123839    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x" event={"ID":"ce76d297-042d-4fb1-8f79-dbe1ccfb546f","Type":"ContainerStarted","Data":"6a0886aa96ca2dd6f177835d84ae28436237455ac20400658f3cfa671323295a"}
Sep 18 18:55:39.124307 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:39.123887    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x" event={"ID":"ce76d297-042d-4fb1-8f79-dbe1ccfb546f","Type":"ContainerStarted","Data":"c3007013291cb86f197af3d800a2ee505f3d63698ba0b548a1fe8c2bcd7c8b52"}
Sep 18 18:55:39.142905 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:39.142858    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x" podStartSLOduration=2.142843379 podStartE2EDuration="2.142843379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:55:39.139328007 +0000 UTC m=+998.575906850" watchObservedRunningTime="2026-09-18 18:55:39.142843379 +0000 UTC m=+998.579422247"
Sep 18 18:55:48.148199 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:48.148150    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:48.148660 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:48.148220    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:48.161558 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:48.161532    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:48.193906 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:48.193881    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 18:55:56.883878 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:56.883839    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w"]
Sep 18 18:55:56.887569 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:56.887553    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w"
Sep 18 18:55:56.889706 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:56.889688    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 18:55:56.889782 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:56.889695    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 18:55:56.894375 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:56.894351    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w"]
Sep 18 18:55:56.999998 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:56.999963    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f18783-f770-40e1-9731-5c7960c742bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w\" (UID: \"48f18783-f770-40e1-9731-5c7960c742bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w"
Sep 18 18:55:57.101374 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:57.101334    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f18783-f770-40e1-9731-5c7960c742bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w\" (UID: \"48f18783-f770-40e1-9731-5c7960c742bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w"
Sep 18 18:55:57.101974 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:57.101956    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f18783-f770-40e1-9731-5c7960c742bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w\" (UID: \"48f18783-f770-40e1-9731-5c7960c742bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w"
Sep 18 18:55:57.200650 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:57.200566    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w"
Sep 18 18:55:57.480540 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:57.480514    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w"]
Sep 18 18:55:57.482334 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:55:57.482310    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f18783_f770_40e1_9731_5c7960c742bb.slice/crio-608cbf85bfb52905fbe117e3aaaa9d48feda40c5bbf558b12e3a3c1e1f55c853 WatchSource:0}: Error finding container 608cbf85bfb52905fbe117e3aaaa9d48feda40c5bbf558b12e3a3c1e1f55c853: Status 404 returned error can't find the container with id 608cbf85bfb52905fbe117e3aaaa9d48feda40c5bbf558b12e3a3c1e1f55c853
Sep 18 18:55:58.255339 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:58.255299    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w" event={"ID":"48f18783-f770-40e1-9731-5c7960c742bb","Type":"ContainerStarted","Data":"f8d90a601d3b32bd09f4bf7ef7f8b30587b00612abed4db251bebb74ebf95da3"}
Sep 18 18:55:58.255694 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:58.255404    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w" event={"ID":"48f18783-f770-40e1-9731-5c7960c742bb","Type":"ContainerStarted","Data":"608cbf85bfb52905fbe117e3aaaa9d48feda40c5bbf558b12e3a3c1e1f55c853"}
Sep 18 18:55:58.281036 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:58.280988    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w" podStartSLOduration=2.280972146 podStartE2EDuration="2.280972146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:55:58.279117151 +0000 UTC m=+1017.715696007" watchObservedRunningTime="2026-09-18 18:55:58.280972146 +0000 UTC m=+1017.717550987"
Sep 18 18:55:59.263984 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:59.263948    2558 generic.go:356] "Generic (PLEG): container finished" podID="48f18783-f770-40e1-9731-5c7960c742bb" containerID="f8d90a601d3b32bd09f4bf7ef7f8b30587b00612abed4db251bebb74ebf95da3" exitCode=0
Sep 18 18:55:59.264390 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:55:59.264036    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w" event={"ID":"48f18783-f770-40e1-9731-5c7960c742bb","Type":"ContainerDied","Data":"f8d90a601d3b32bd09f4bf7ef7f8b30587b00612abed4db251bebb74ebf95da3"}
Sep 18 18:56:00.415530 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:56:00.415508    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w"
Sep 18 18:56:00.534390 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:56:00.534309    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f18783-f770-40e1-9731-5c7960c742bb-maas-api-service-ca\") pod \"48f18783-f770-40e1-9731-5c7960c742bb\" (UID: \"48f18783-f770-40e1-9731-5c7960c742bb\") "
Sep 18 18:56:00.534743 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:56:00.534718    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f18783-f770-40e1-9731-5c7960c742bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48f18783-f770-40e1-9731-5c7960c742bb" (UID: "48f18783-f770-40e1-9731-5c7960c742bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:56:00.635920 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:56:00.635894    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48f18783-f770-40e1-9731-5c7960c742bb-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:56:01.277847 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:56:01.277815    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w"
Sep 18 18:56:01.277847 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:56:01.277840    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w" event={"ID":"48f18783-f770-40e1-9731-5c7960c742bb","Type":"ContainerDied","Data":"608cbf85bfb52905fbe117e3aaaa9d48feda40c5bbf558b12e3a3c1e1f55c853"}
Sep 18 18:56:01.278069 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:56:01.277887    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="608cbf85bfb52905fbe117e3aaaa9d48feda40c5bbf558b12e3a3c1e1f55c853"
Sep 18 18:56:04.576557 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:56:04.576523    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w"]
Sep 18 18:56:04.576557 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:56:04.576563    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d2dcf91d-rjz4w"]
Sep 18 18:56:06.575933 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:56:06.575899    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48f18783-f770-40e1-9731-5c7960c742bb" path="/var/lib/kubelet/pods/48f18783-f770-40e1-9731-5c7960c742bb/volumes"
Sep 18 18:57:40.808744 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.808705    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"]
Sep 18 18:57:40.809287 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.809119    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f18783-f770-40e1-9731-5c7960c742bb" containerName="revoke-keys"
Sep 18 18:57:40.809287 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.809130    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f18783-f770-40e1-9731-5c7960c742bb" containerName="revoke-keys"
Sep 18 18:57:40.809287 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.809238    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="48f18783-f770-40e1-9731-5c7960c742bb" containerName="revoke-keys"
Sep 18 18:57:40.812413 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.812392    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:40.814903 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.814881    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-62de055a-openshift-default-dockercfg-bttds\""
Sep 18 18:57:40.823528 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.823508    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"]
Sep 18 18:57:40.950696 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.950592    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc778\" (UniqueName: \"kubernetes.io/projected/1f30ea4f-c89e-417a-bd84-c2e807855e37-kube-api-access-jc778\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:40.950696 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.950655    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-workload-certs\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:40.950893 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.950730    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-data\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:40.950893 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.950771    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f30ea4f-c89e-417a-bd84-c2e807855e37-istiod-ca-cert\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:40.950893 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.950803    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-token\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:40.950893 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.950834    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-workload-socket\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:40.950893 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.950859    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-credential-socket\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:40.951055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.950916    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-envoy\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:40.951055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:40.950953    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-podinfo\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.052263 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.052231    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc778\" (UniqueName: \"kubernetes.io/projected/1f30ea4f-c89e-417a-bd84-c2e807855e37-kube-api-access-jc778\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.052462 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.052278    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-workload-certs\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.052462 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.052358    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-data\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.052462 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.052400    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f30ea4f-c89e-417a-bd84-c2e807855e37-istiod-ca-cert\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.052462 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.052433    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-token\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.052685 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.052464    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-workload-socket\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.052685 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.052490    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-credential-socket\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.052685 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.052520    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-envoy\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.052685 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.052566    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-podinfo\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.052882 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.052694    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-workload-certs\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.053594 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.053297    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-credential-socket\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.053594 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.053323    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-data\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.053791 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.053596    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-workload-socket\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.053791 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.053726    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f30ea4f-c89e-417a-bd84-c2e807855e37-istiod-ca-cert\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.055349 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.055327    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-podinfo\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.055568 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.055550    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-envoy\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.060894 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.060867    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-token\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.061171 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.061149    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc778\" (UniqueName: \"kubernetes.io/projected/1f30ea4f-c89e-417a-bd84-c2e807855e37-kube-api-access-jc778\") pod \"e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") " pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.124722 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.124689    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:41.407088 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.407055    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"]
Sep 18 18:57:41.408123 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:57:41.408092    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f30ea4f_c89e_417a_bd84_c2e807855e37.slice/crio-44a4fb71eba15d7131b624767c089207d9c84fc4eee3e44a386e5b1156c21377 WatchSource:0}: Error finding container 44a4fb71eba15d7131b624767c089207d9c84fc4eee3e44a386e5b1156c21377: Status 404 returned error can't find the container with id 44a4fb71eba15d7131b624767c089207d9c84fc4eee3e44a386e5b1156c21377
Sep 18 18:57:41.410389 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.410354    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:57:41.410458 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.410423    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:57:41.410496 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.410455    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:57:41.940824 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.940788    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl" event={"ID":"1f30ea4f-c89e-417a-bd84-c2e807855e37","Type":"ContainerStarted","Data":"2cd53ca63ba376ef66f3527ff980638c5b7c45418cdcfdec57d48e1d1d76e21d"}
Sep 18 18:57:41.941229 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.940844    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl" event={"ID":"1f30ea4f-c89e-417a-bd84-c2e807855e37","Type":"ContainerStarted","Data":"44a4fb71eba15d7131b624767c089207d9c84fc4eee3e44a386e5b1156c21377"}
Sep 18 18:57:41.962809 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:41.962759    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl" podStartSLOduration=1.9627438750000001 podStartE2EDuration="1.962743875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:57:41.959684723 +0000 UTC m=+1121.396263587" watchObservedRunningTime="2026-09-18 18:57:41.962743875 +0000 UTC m=+1121.399322718"
Sep 18 18:57:42.125358 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:42.125324    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:42.130439 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:42.130413    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:42.946785 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:42.946754    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:57:42.947979 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:57:42.947964    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:58:03.312099 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:03.312065    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"]
Sep 18 18:58:05.259759 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:05.259721    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf"]
Sep 18 18:58:05.263472 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:05.263455    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf"
Sep 18 18:58:05.265539 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:05.265516    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 18:58:05.265779 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:05.265763    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 18:58:05.272758 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:05.272734    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf"]
Sep 18 18:58:05.296555 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:05.296529    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e350ac1-5735-48b4-a410-bb5098876477-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf\" (UID: \"5e350ac1-5735-48b4-a410-bb5098876477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf"
Sep 18 18:58:05.397398 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:05.397369    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e350ac1-5735-48b4-a410-bb5098876477-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf\" (UID: \"5e350ac1-5735-48b4-a410-bb5098876477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf"
Sep 18 18:58:05.397972 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:05.397953    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e350ac1-5735-48b4-a410-bb5098876477-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf\" (UID: \"5e350ac1-5735-48b4-a410-bb5098876477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf"
Sep 18 18:58:05.575978 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:05.575898    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf"
Sep 18 18:58:05.856571 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:05.856535    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf"]
Sep 18 18:58:05.857579 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:58:05.857555    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e350ac1_5735_48b4_a410_bb5098876477.slice/crio-35b34ec0290b2dfc28a38b46a7d7191f9ed319cad400b0754125d0561ce4e94e WatchSource:0}: Error finding container 35b34ec0290b2dfc28a38b46a7d7191f9ed319cad400b0754125d0561ce4e94e: Status 404 returned error can't find the container with id 35b34ec0290b2dfc28a38b46a7d7191f9ed319cad400b0754125d0561ce4e94e
Sep 18 18:58:06.100245 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:06.100213    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf" event={"ID":"5e350ac1-5735-48b4-a410-bb5098876477","Type":"ContainerStarted","Data":"35b34ec0290b2dfc28a38b46a7d7191f9ed319cad400b0754125d0561ce4e94e"}
Sep 18 18:58:07.109914 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.109882    2558 generic.go:356] "Generic (PLEG): container finished" podID="5e350ac1-5735-48b4-a410-bb5098876477" containerID="61ac10ab5fb6557d50152f496bcb0021829c6403f49915f2cf893d19fc162871" exitCode=0
Sep 18 18:58:07.110311 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.109971    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf" event={"ID":"5e350ac1-5735-48b4-a410-bb5098876477","Type":"ContainerDied","Data":"61ac10ab5fb6557d50152f496bcb0021829c6403f49915f2cf893d19fc162871"}
Sep 18 18:58:07.236410 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.236378    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"]
Sep 18 18:58:07.240067 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.240044    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.242387 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.242363    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-cc289c\"/\"distinct-2-w0-cc289c-kserve-self-signed-certs\""
Sep 18 18:58:07.251677 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.251649    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"]
Sep 18 18:58:07.325934 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.325901    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-tmp-dir\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.325934 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.325934    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-home\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.326139 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.325969    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-model-cache\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.326139 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.326036    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8ffd\" (UniqueName: \"kubernetes.io/projected/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-kube-api-access-h8ffd\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.326139 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.326089    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-tls-certs\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.326278 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.326174    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-dshm\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.427295 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.427168    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-model-cache\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.427295 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.427251    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8ffd\" (UniqueName: \"kubernetes.io/projected/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-kube-api-access-h8ffd\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.427295 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.427297    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-tls-certs\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.427562 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.427355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-dshm\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.427562 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.427507    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-tmp-dir\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.427656 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.427566    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-home\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.427704 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.427565    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-model-cache\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.427822 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.427803    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-home\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.427862 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.427811    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-tmp-dir\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.429673 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.429653    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-dshm\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.429854 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.429837    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-tls-certs\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.434553 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.434534    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8ffd\" (UniqueName: \"kubernetes.io/projected/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-kube-api-access-h8ffd\") pod \"distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") " pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.551038 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.551007    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:07.825888 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:07.825860    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"]
Sep 18 18:58:07.828063 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:58:07.828036    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ae3183f_87f3_4891_8aef_7f85a6db3ccf.slice/crio-1f046baea2c1927ca45ca1b7c938234b87ac8dfb7a9d2048e4e0e4e24014535f WatchSource:0}: Error finding container 1f046baea2c1927ca45ca1b7c938234b87ac8dfb7a9d2048e4e0e4e24014535f: Status 404 returned error can't find the container with id 1f046baea2c1927ca45ca1b7c938234b87ac8dfb7a9d2048e4e0e4e24014535f
Sep 18 18:58:08.118424 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:08.118385    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv" event={"ID":"9ae3183f-87f3-4891-8aef-7f85a6db3ccf","Type":"ContainerStarted","Data":"df0cf48996456c3450eb512115435814c8efe7fd914a120cf9a6674c53df238d"}
Sep 18 18:58:08.118822 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:08.118437    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv" event={"ID":"9ae3183f-87f3-4891-8aef-7f85a6db3ccf","Type":"ContainerStarted","Data":"1f046baea2c1927ca45ca1b7c938234b87ac8dfb7a9d2048e4e0e4e24014535f"}
Sep 18 18:58:08.134761 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:08.134698    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv" podStartSLOduration=1.134676054 podStartE2EDuration="1.134676054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:58:08.132924005 +0000 UTC m=+1147.569502849" watchObservedRunningTime="2026-09-18 18:58:08.134676054 +0000 UTC m=+1147.571254973"
Sep 18 18:58:08.253763 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:08.253244    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf"
Sep 18 18:58:08.337375 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:08.337341    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e350ac1-5735-48b4-a410-bb5098876477-maas-api-service-ca\") pod \"5e350ac1-5735-48b4-a410-bb5098876477\" (UID: \"5e350ac1-5735-48b4-a410-bb5098876477\") "
Sep 18 18:58:08.337742 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:08.337721    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e350ac1-5735-48b4-a410-bb5098876477-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e350ac1-5735-48b4-a410-bb5098876477" (UID: "5e350ac1-5735-48b4-a410-bb5098876477"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:58:08.438753 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:08.438713    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e350ac1-5735-48b4-a410-bb5098876477-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:09.126792 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:09.126758    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf"
Sep 18 18:58:09.127198 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:09.126788    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf" event={"ID":"5e350ac1-5735-48b4-a410-bb5098876477","Type":"ContainerDied","Data":"35b34ec0290b2dfc28a38b46a7d7191f9ed319cad400b0754125d0561ce4e94e"}
Sep 18 18:58:09.127198 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:09.126875    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b34ec0290b2dfc28a38b46a7d7191f9ed319cad400b0754125d0561ce4e94e"
Sep 18 18:58:16.241175 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:16.241086    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf"]
Sep 18 18:58:16.245875 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:16.245849    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bcb770-grcwf"]
Sep 18 18:58:16.576946 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:16.576872    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e350ac1-5735-48b4-a410-bb5098876477" path="/var/lib/kubelet/pods/5e350ac1-5735-48b4-a410-bb5098876477/volumes"
Sep 18 18:58:17.551296 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:17.551251    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:17.551725 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:17.551315    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:17.564614 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:17.564585    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:18.201381 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.201294    2558 generic.go:356] "Generic (PLEG): container finished" podID="1f30ea4f-c89e-417a-bd84-c2e807855e37" containerID="2cd53ca63ba376ef66f3527ff980638c5b7c45418cdcfdec57d48e1d1d76e21d" exitCode=0
Sep 18 18:58:18.201561 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.201375    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl" event={"ID":"1f30ea4f-c89e-417a-bd84-c2e807855e37","Type":"ContainerDied","Data":"2cd53ca63ba376ef66f3527ff980638c5b7c45418cdcfdec57d48e1d1d76e21d"}
Sep 18 18:58:18.201963 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.201937    2558 scope.go:117] "RemoveContainer" containerID="2cd53ca63ba376ef66f3527ff980638c5b7c45418cdcfdec57d48e1d1d76e21d"
Sep 18 18:58:18.203206 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.203149    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:58:18.203309 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.203262    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:58:18.203309 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.203298    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:58:18.215247 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.215227    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 18:58:18.331942 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.331901    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586" podUID="f3dd0b58-1324-4295-9425-8990c8959583" containerName="main" containerID="cri-o://86cb1e5593c1bb158cb3dee17d25fb62161b0caba642a6feb93e477e96572cc3" gracePeriod=45
Sep 18 18:58:18.602788 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.602762    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:58:18.760580 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.760535    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-tmp-dir\") pod \"f3dd0b58-1324-4295-9425-8990c8959583\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") "
Sep 18 18:58:18.760922 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.760902    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-model-cache\") pod \"f3dd0b58-1324-4295-9425-8990c8959583\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") "
Sep 18 18:58:18.761065 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.761049    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qt92d\" (UniqueName: \"kubernetes.io/projected/f3dd0b58-1324-4295-9425-8990c8959583-kube-api-access-qt92d\") pod \"f3dd0b58-1324-4295-9425-8990c8959583\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") "
Sep 18 18:58:18.761290 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.761275    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3dd0b58-1324-4295-9425-8990c8959583-tls-certs\") pod \"f3dd0b58-1324-4295-9425-8990c8959583\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") "
Sep 18 18:58:18.761482 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.761468    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-dshm\") pod \"f3dd0b58-1324-4295-9425-8990c8959583\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") "
Sep 18 18:58:18.761636 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.761616    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-home\") pod \"f3dd0b58-1324-4295-9425-8990c8959583\" (UID: \"f3dd0b58-1324-4295-9425-8990c8959583\") "
Sep 18 18:58:18.761759 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.761056    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3dd0b58-1324-4295-9425-8990c8959583" (UID: "f3dd0b58-1324-4295-9425-8990c8959583"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:58:18.762095 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.761177    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3dd0b58-1324-4295-9425-8990c8959583" (UID: "f3dd0b58-1324-4295-9425-8990c8959583"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:58:18.762208 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.762037    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-home" (OuterVolumeSpecName: "home") pod "f3dd0b58-1324-4295-9425-8990c8959583" (UID: "f3dd0b58-1324-4295-9425-8990c8959583"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:58:18.762449 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.762428    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-home\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:18.762559 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.762547    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-tmp-dir\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:18.762644 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.762632    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-model-cache\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:18.768387 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.768296    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-dshm" (OuterVolumeSpecName: "dshm") pod "f3dd0b58-1324-4295-9425-8990c8959583" (UID: "f3dd0b58-1324-4295-9425-8990c8959583"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:58:18.771392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.771344    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3dd0b58-1324-4295-9425-8990c8959583-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3dd0b58-1324-4295-9425-8990c8959583" (UID: "f3dd0b58-1324-4295-9425-8990c8959583"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:58:18.771392 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.771354    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3dd0b58-1324-4295-9425-8990c8959583-kube-api-access-qt92d" (OuterVolumeSpecName: "kube-api-access-qt92d") pod "f3dd0b58-1324-4295-9425-8990c8959583" (UID: "f3dd0b58-1324-4295-9425-8990c8959583"). InnerVolumeSpecName "kube-api-access-qt92d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:58:18.863554 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.863463    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt92d\" (UniqueName: \"kubernetes.io/projected/f3dd0b58-1324-4295-9425-8990c8959583-kube-api-access-qt92d\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:18.863554 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.863510    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3dd0b58-1324-4295-9425-8990c8959583-tls-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:18.863554 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:18.863526    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3dd0b58-1324-4295-9425-8990c8959583-dshm\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:19.209493 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:19.209459    2558 generic.go:356] "Generic (PLEG): container finished" podID="f3dd0b58-1324-4295-9425-8990c8959583" containerID="86cb1e5593c1bb158cb3dee17d25fb62161b0caba642a6feb93e477e96572cc3" exitCode=0
Sep 18 18:58:19.209719 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:19.209513    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"
Sep 18 18:58:19.209719 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:19.209550    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586" event={"ID":"f3dd0b58-1324-4295-9425-8990c8959583","Type":"ContainerDied","Data":"86cb1e5593c1bb158cb3dee17d25fb62161b0caba642a6feb93e477e96572cc3"}
Sep 18 18:58:19.209719 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:19.209594    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586" event={"ID":"f3dd0b58-1324-4295-9425-8990c8959583","Type":"ContainerDied","Data":"8443b37606377f3b439ef964b1ba7710883b2c20a30c45ced6998a4ca1797085"}
Sep 18 18:58:19.209719 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:19.209617    2558 scope.go:117] "RemoveContainer" containerID="86cb1e5593c1bb158cb3dee17d25fb62161b0caba642a6feb93e477e96572cc3"
Sep 18 18:58:19.212107 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:19.212072    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl" event={"ID":"1f30ea4f-c89e-417a-bd84-c2e807855e37","Type":"ContainerStarted","Data":"2b1f5646a853cc917f1eed78dfe3135f1134713f5c06b75e9f59e508075a388c"}
Sep 18 18:58:19.220656 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:19.220634    2558 scope.go:117] "RemoveContainer" containerID="86cb1e5593c1bb158cb3dee17d25fb62161b0caba642a6feb93e477e96572cc3"
Sep 18 18:58:19.220919 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:58:19.220901    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86cb1e5593c1bb158cb3dee17d25fb62161b0caba642a6feb93e477e96572cc3\": container with ID starting with 86cb1e5593c1bb158cb3dee17d25fb62161b0caba642a6feb93e477e96572cc3 not found: ID does not exist" containerID="86cb1e5593c1bb158cb3dee17d25fb62161b0caba642a6feb93e477e96572cc3"
Sep 18 18:58:19.220996 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:19.220933    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86cb1e5593c1bb158cb3dee17d25fb62161b0caba642a6feb93e477e96572cc3"} err="failed to get container status \"86cb1e5593c1bb158cb3dee17d25fb62161b0caba642a6feb93e477e96572cc3\": rpc error: code = NotFound desc = could not find container \"86cb1e5593c1bb158cb3dee17d25fb62161b0caba642a6feb93e477e96572cc3\": container with ID starting with 86cb1e5593c1bb158cb3dee17d25fb62161b0caba642a6feb93e477e96572cc3 not found: ID does not exist"
Sep 18 18:58:19.242589 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:19.242563    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"]
Sep 18 18:58:19.246176 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:19.246151    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bcb770/test-model-bcb770-kserve-564498dc-fb586"]
Sep 18 18:58:20.125035 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:20.125003    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:58:20.125459 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:20.125178    2558 patch_prober.go:28] interesting pod/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 18 18:58:20.125459 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:20.125239    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl" podUID="1f30ea4f-c89e-417a-bd84-c2e807855e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 18 18:58:20.579550 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:20.579469    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3dd0b58-1324-4295-9425-8990c8959583" path="/var/lib/kubelet/pods/f3dd0b58-1324-4295-9425-8990c8959583/volumes"
Sep 18 18:58:21.002757 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.002713    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg"]
Sep 18 18:58:21.003240 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.003174    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3dd0b58-1324-4295-9425-8990c8959583" containerName="main"
Sep 18 18:58:21.003240 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.003202    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3dd0b58-1324-4295-9425-8990c8959583" containerName="main"
Sep 18 18:58:21.003240 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.003218    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e350ac1-5735-48b4-a410-bb5098876477" containerName="revoke-keys"
Sep 18 18:58:21.003240 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.003224    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e350ac1-5735-48b4-a410-bb5098876477" containerName="revoke-keys"
Sep 18 18:58:21.003467 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.003320    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e350ac1-5735-48b4-a410-bb5098876477" containerName="revoke-keys"
Sep 18 18:58:21.003467 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.003333    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3dd0b58-1324-4295-9425-8990c8959583" containerName="main"
Sep 18 18:58:21.007999 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.007982    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg"
Sep 18 18:58:21.010722 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.010509    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 18:58:21.010722 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.010675    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 18:58:21.013468 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.013444    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg"]
Sep 18 18:58:21.125735 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.125703    2558 patch_prober.go:28] interesting pod/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 18 18:58:21.126126 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.125765    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl" podUID="1f30ea4f-c89e-417a-bd84-c2e807855e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 18 18:58:21.190535 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.190496    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30bb7138-5200-4731-ab2b-637bfacd57eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-62de055a-mqxgg\" (UID: \"30bb7138-5200-4731-ab2b-637bfacd57eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg"
Sep 18 18:58:21.291543 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.291449    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30bb7138-5200-4731-ab2b-637bfacd57eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-62de055a-mqxgg\" (UID: \"30bb7138-5200-4731-ab2b-637bfacd57eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg"
Sep 18 18:58:21.292104 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.292083    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30bb7138-5200-4731-ab2b-637bfacd57eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-62de055a-mqxgg\" (UID: \"30bb7138-5200-4731-ab2b-637bfacd57eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg"
Sep 18 18:58:21.321207 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.321155    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg"
Sep 18 18:58:21.619023 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:21.618997    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg"]
Sep 18 18:58:21.621164 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:58:21.621133    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bb7138_5200_4731_ab2b_637bfacd57eb.slice/crio-f87d5ee0470a56de5037fb381430a35d61e09ee67f17941f3c112c7707960b72 WatchSource:0}: Error finding container f87d5ee0470a56de5037fb381430a35d61e09ee67f17941f3c112c7707960b72: Status 404 returned error can't find the container with id f87d5ee0470a56de5037fb381430a35d61e09ee67f17941f3c112c7707960b72
Sep 18 18:58:22.125426 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:22.125392    2558 patch_prober.go:28] interesting pod/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 18 18:58:22.125574 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:22.125456    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl" podUID="1f30ea4f-c89e-417a-bd84-c2e807855e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 18 18:58:22.235296 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:22.235174    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg" event={"ID":"30bb7138-5200-4731-ab2b-637bfacd57eb","Type":"ContainerStarted","Data":"53b4971c344a8ae10839cc0031e27ed66dcde9e712e55f2d8cbfb74d69cd5d95"}
Sep 18 18:58:22.235296 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:22.235261    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg" event={"ID":"30bb7138-5200-4731-ab2b-637bfacd57eb","Type":"ContainerStarted","Data":"f87d5ee0470a56de5037fb381430a35d61e09ee67f17941f3c112c7707960b72"}
Sep 18 18:58:22.251010 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:22.250946    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg" podStartSLOduration=2.250925581 podStartE2EDuration="2.250925581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:58:22.250163956 +0000 UTC m=+1161.686742799" watchObservedRunningTime="2026-09-18 18:58:22.250925581 +0000 UTC m=+1161.687504426"
Sep 18 18:58:23.125899 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:23.125867    2558 patch_prober.go:28] interesting pod/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 18 18:58:23.126078 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:23.125923    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl" podUID="1f30ea4f-c89e-417a-bd84-c2e807855e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 18 18:58:23.243753 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:23.243715    2558 generic.go:356] "Generic (PLEG): container finished" podID="30bb7138-5200-4731-ab2b-637bfacd57eb" containerID="53b4971c344a8ae10839cc0031e27ed66dcde9e712e55f2d8cbfb74d69cd5d95" exitCode=0
Sep 18 18:58:23.244261 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:23.243805    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg" event={"ID":"30bb7138-5200-4731-ab2b-637bfacd57eb","Type":"ContainerDied","Data":"53b4971c344a8ae10839cc0031e27ed66dcde9e712e55f2d8cbfb74d69cd5d95"}
Sep 18 18:58:23.390050 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:23.389945    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl"]
Sep 18 18:58:23.393879 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:23.393854    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl"
Sep 18 18:58:23.403918 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:23.403892    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl"]
Sep 18 18:58:23.515355 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:23.515320    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74bd161d-4fb3-4a8c-8ecb-5437e55b4842-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl\" (UID: \"74bd161d-4fb3-4a8c-8ecb-5437e55b4842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl"
Sep 18 18:58:23.616856 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:23.616824    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74bd161d-4fb3-4a8c-8ecb-5437e55b4842-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl\" (UID: \"74bd161d-4fb3-4a8c-8ecb-5437e55b4842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl"
Sep 18 18:58:23.617542 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:23.617521    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74bd161d-4fb3-4a8c-8ecb-5437e55b4842-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl\" (UID: \"74bd161d-4fb3-4a8c-8ecb-5437e55b4842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl"
Sep 18 18:58:23.706163 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:23.706075    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl"
Sep 18 18:58:24.011626 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:24.011597    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl"]
Sep 18 18:58:24.013345 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:58:24.013319    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bd161d_4fb3_4a8c_8ecb_5437e55b4842.slice/crio-adbffb2164d3b77f452f7bd57b1fad4c0f15ffe66188db0049eef04828f9a18a WatchSource:0}: Error finding container adbffb2164d3b77f452f7bd57b1fad4c0f15ffe66188db0049eef04828f9a18a: Status 404 returned error can't find the container with id adbffb2164d3b77f452f7bd57b1fad4c0f15ffe66188db0049eef04828f9a18a
Sep 18 18:58:24.126199 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:24.126164    2558 patch_prober.go:28] interesting pod/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 18 18:58:24.126344 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:24.126245    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl" podUID="1f30ea4f-c89e-417a-bd84-c2e807855e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 18 18:58:24.251980 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:24.251938    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl" event={"ID":"74bd161d-4fb3-4a8c-8ecb-5437e55b4842","Type":"ContainerStarted","Data":"adbffb2164d3b77f452f7bd57b1fad4c0f15ffe66188db0049eef04828f9a18a"}
Sep 18 18:58:24.397568 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:24.397546    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg"
Sep 18 18:58:24.526004 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:24.525971    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30bb7138-5200-4731-ab2b-637bfacd57eb-maas-api-service-ca\") pod \"30bb7138-5200-4731-ab2b-637bfacd57eb\" (UID: \"30bb7138-5200-4731-ab2b-637bfacd57eb\") "
Sep 18 18:58:24.526369 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:24.526344    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bb7138-5200-4731-ab2b-637bfacd57eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "30bb7138-5200-4731-ab2b-637bfacd57eb" (UID: "30bb7138-5200-4731-ab2b-637bfacd57eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:58:24.627772 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:24.627742    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30bb7138-5200-4731-ab2b-637bfacd57eb-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:25.130133 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:25.130031    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:58:25.130439 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:25.130407    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:58:25.131062 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:25.131046    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:58:25.260029 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:25.260002    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg"
Sep 18 18:58:25.260486 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:25.260038    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg" event={"ID":"30bb7138-5200-4731-ab2b-637bfacd57eb","Type":"ContainerDied","Data":"f87d5ee0470a56de5037fb381430a35d61e09ee67f17941f3c112c7707960b72"}
Sep 18 18:58:25.260486 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:25.260091    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f87d5ee0470a56de5037fb381430a35d61e09ee67f17941f3c112c7707960b72"
Sep 18 18:58:25.261805 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:25.261772    2558 generic.go:356] "Generic (PLEG): container finished" podID="74bd161d-4fb3-4a8c-8ecb-5437e55b4842" containerID="71c9d03e387d71c477ddb518fcc880473a55ce6c43f54d1131e57c1a56ffb3fc" exitCode=0
Sep 18 18:58:25.261948 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:25.261923    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl" event={"ID":"74bd161d-4fb3-4a8c-8ecb-5437e55b4842","Type":"ContainerDied","Data":"71c9d03e387d71c477ddb518fcc880473a55ce6c43f54d1131e57c1a56ffb3fc"}
Sep 18 18:58:26.408705 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:26.408679    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl"
Sep 18 18:58:26.446932 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:26.446895    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74bd161d-4fb3-4a8c-8ecb-5437e55b4842-maas-api-service-ca\") pod \"74bd161d-4fb3-4a8c-8ecb-5437e55b4842\" (UID: \"74bd161d-4fb3-4a8c-8ecb-5437e55b4842\") "
Sep 18 18:58:26.447310 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:26.447287    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bd161d-4fb3-4a8c-8ecb-5437e55b4842-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74bd161d-4fb3-4a8c-8ecb-5437e55b4842" (UID: "74bd161d-4fb3-4a8c-8ecb-5437e55b4842"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:58:26.547875 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:26.547780    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74bd161d-4fb3-4a8c-8ecb-5437e55b4842-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:27.275680 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:27.275648    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl"
Sep 18 18:58:27.275680 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:27.275669    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl" event={"ID":"74bd161d-4fb3-4a8c-8ecb-5437e55b4842","Type":"ContainerDied","Data":"adbffb2164d3b77f452f7bd57b1fad4c0f15ffe66188db0049eef04828f9a18a"}
Sep 18 18:58:27.275945 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:27.275737    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adbffb2164d3b77f452f7bd57b1fad4c0f15ffe66188db0049eef04828f9a18a"
Sep 18 18:58:32.066770 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:32.066737    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg"]
Sep 18 18:58:32.072594 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:32.072564    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62de055a-mqxgg"]
Sep 18 18:58:32.580250 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:32.580207    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30bb7138-5200-4731-ab2b-637bfacd57eb" path="/var/lib/kubelet/pods/30bb7138-5200-4731-ab2b-637bfacd57eb/volumes"
Sep 18 18:58:34.662514 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:34.662477    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl"]
Sep 18 18:58:34.665208 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:34.665172    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-897f61-f4mfl"]
Sep 18 18:58:36.576348 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:36.576305    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74bd161d-4fb3-4a8c-8ecb-5437e55b4842" path="/var/lib/kubelet/pods/74bd161d-4fb3-4a8c-8ecb-5437e55b4842/volumes"
Sep 18 18:58:38.767705 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:38.767660    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"]
Sep 18 18:58:38.768147 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:38.767954    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl" podUID="1f30ea4f-c89e-417a-bd84-c2e807855e37" containerName="istio-proxy" containerID="cri-o://2b1f5646a853cc917f1eed78dfe3135f1134713f5c06b75e9f59e508075a388c" gracePeriod=30
Sep 18 18:58:40.132550 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:40.132509    2558 patch_prober.go:28] interesting pod/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 18 18:58:40.132930 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:40.132584    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl" podUID="1f30ea4f-c89e-417a-bd84-c2e807855e37" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 18:58:41.146128 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:41.145096    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"]
Sep 18 18:58:41.146128 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:41.145434    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch" podUID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerName="istio-proxy" containerID="cri-o://99a40c27ebeab44776884112a4fd42c98949273c227483d0e73919b34933d30c" gracePeriod=30
Sep 18 18:58:44.023302 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.023278    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:58:44.146055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.146023    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-podinfo\") pod \"1f30ea4f-c89e-417a-bd84-c2e807855e37\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") "
Sep 18 18:58:44.146271 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.146087    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-credential-socket\") pod \"1f30ea4f-c89e-417a-bd84-c2e807855e37\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") "
Sep 18 18:58:44.146271 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.146110    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-envoy\") pod \"1f30ea4f-c89e-417a-bd84-c2e807855e37\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") "
Sep 18 18:58:44.146271 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.146175    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc778\" (UniqueName: \"kubernetes.io/projected/1f30ea4f-c89e-417a-bd84-c2e807855e37-kube-api-access-jc778\") pod \"1f30ea4f-c89e-417a-bd84-c2e807855e37\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") "
Sep 18 18:58:44.146457 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.146277    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-workload-certs\") pod \"1f30ea4f-c89e-417a-bd84-c2e807855e37\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") "
Sep 18 18:58:44.146457 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.146316    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-data\") pod \"1f30ea4f-c89e-417a-bd84-c2e807855e37\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") "
Sep 18 18:58:44.146457 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.146338    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-token\") pod \"1f30ea4f-c89e-417a-bd84-c2e807855e37\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") "
Sep 18 18:58:44.146457 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.146401    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f30ea4f-c89e-417a-bd84-c2e807855e37-istiod-ca-cert\") pod \"1f30ea4f-c89e-417a-bd84-c2e807855e37\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") "
Sep 18 18:58:44.146457 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.146445    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-workload-socket\") pod \"1f30ea4f-c89e-417a-bd84-c2e807855e37\" (UID: \"1f30ea4f-c89e-417a-bd84-c2e807855e37\") "
Sep 18 18:58:44.146717 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.146472    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1f30ea4f-c89e-417a-bd84-c2e807855e37" (UID: "1f30ea4f-c89e-417a-bd84-c2e807855e37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:58:44.146717 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.146698    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-data" (OuterVolumeSpecName: "istio-data") pod "1f30ea4f-c89e-417a-bd84-c2e807855e37" (UID: "1f30ea4f-c89e-417a-bd84-c2e807855e37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:58:44.146943 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.146914    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1f30ea4f-c89e-417a-bd84-c2e807855e37" (UID: "1f30ea4f-c89e-417a-bd84-c2e807855e37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:58:44.147314 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.147257    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1f30ea4f-c89e-417a-bd84-c2e807855e37" (UID: "1f30ea4f-c89e-417a-bd84-c2e807855e37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:58:44.147523 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.147502    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:44.147675 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.147587    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:44.147675 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.147537    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f30ea4f-c89e-417a-bd84-c2e807855e37-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1f30ea4f-c89e-417a-bd84-c2e807855e37" (UID: "1f30ea4f-c89e-417a-bd84-c2e807855e37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:58:44.148803 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.148760    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f30ea4f-c89e-417a-bd84-c2e807855e37-kube-api-access-jc778" (OuterVolumeSpecName: "kube-api-access-jc778") pod "1f30ea4f-c89e-417a-bd84-c2e807855e37" (UID: "1f30ea4f-c89e-417a-bd84-c2e807855e37"). InnerVolumeSpecName "kube-api-access-jc778". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:58:44.148917 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.148877    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1f30ea4f-c89e-417a-bd84-c2e807855e37" (UID: "1f30ea4f-c89e-417a-bd84-c2e807855e37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:58:44.148970 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.148957    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1f30ea4f-c89e-417a-bd84-c2e807855e37" (UID: "1f30ea4f-c89e-417a-bd84-c2e807855e37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:58:44.149328 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.149304    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-token" (OuterVolumeSpecName: "istio-token") pod "1f30ea4f-c89e-417a-bd84-c2e807855e37" (UID: "1f30ea4f-c89e-417a-bd84-c2e807855e37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:58:44.248664 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.248631    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:44.248664 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.248658    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:44.248664 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.248667    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f30ea4f-c89e-417a-bd84-c2e807855e37-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:44.248894 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.248676    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:44.248894 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.248685    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:44.248894 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.248692    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f30ea4f-c89e-417a-bd84-c2e807855e37-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:44.248894 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.248700    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc778\" (UniqueName: \"kubernetes.io/projected/1f30ea4f-c89e-417a-bd84-c2e807855e37-kube-api-access-jc778\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:44.403139 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.403037    2558 generic.go:356] "Generic (PLEG): container finished" podID="1f30ea4f-c89e-417a-bd84-c2e807855e37" containerID="2b1f5646a853cc917f1eed78dfe3135f1134713f5c06b75e9f59e508075a388c" exitCode=0
Sep 18 18:58:44.403139 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.403117    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"
Sep 18 18:58:44.403139 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.403116    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl" event={"ID":"1f30ea4f-c89e-417a-bd84-c2e807855e37","Type":"ContainerDied","Data":"2b1f5646a853cc917f1eed78dfe3135f1134713f5c06b75e9f59e508075a388c"}
Sep 18 18:58:44.403397 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.403236    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl" event={"ID":"1f30ea4f-c89e-417a-bd84-c2e807855e37","Type":"ContainerDied","Data":"44a4fb71eba15d7131b624767c089207d9c84fc4eee3e44a386e5b1156c21377"}
Sep 18 18:58:44.403397 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.403270    2558 scope.go:117] "RemoveContainer" containerID="2b1f5646a853cc917f1eed78dfe3135f1134713f5c06b75e9f59e508075a388c"
Sep 18 18:58:44.413469 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.413453    2558 scope.go:117] "RemoveContainer" containerID="2cd53ca63ba376ef66f3527ff980638c5b7c45418cdcfdec57d48e1d1d76e21d"
Sep 18 18:58:44.422522 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.422502    2558 scope.go:117] "RemoveContainer" containerID="2b1f5646a853cc917f1eed78dfe3135f1134713f5c06b75e9f59e508075a388c"
Sep 18 18:58:44.422795 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:58:44.422775    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1f5646a853cc917f1eed78dfe3135f1134713f5c06b75e9f59e508075a388c\": container with ID starting with 2b1f5646a853cc917f1eed78dfe3135f1134713f5c06b75e9f59e508075a388c not found: ID does not exist" containerID="2b1f5646a853cc917f1eed78dfe3135f1134713f5c06b75e9f59e508075a388c"
Sep 18 18:58:44.422861 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.422802    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1f5646a853cc917f1eed78dfe3135f1134713f5c06b75e9f59e508075a388c"} err="failed to get container status \"2b1f5646a853cc917f1eed78dfe3135f1134713f5c06b75e9f59e508075a388c\": rpc error: code = NotFound desc = could not find container \"2b1f5646a853cc917f1eed78dfe3135f1134713f5c06b75e9f59e508075a388c\": container with ID starting with 2b1f5646a853cc917f1eed78dfe3135f1134713f5c06b75e9f59e508075a388c not found: ID does not exist"
Sep 18 18:58:44.422861 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.422818    2558 scope.go:117] "RemoveContainer" containerID="2cd53ca63ba376ef66f3527ff980638c5b7c45418cdcfdec57d48e1d1d76e21d"
Sep 18 18:58:44.423130 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:58:44.423114    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd53ca63ba376ef66f3527ff980638c5b7c45418cdcfdec57d48e1d1d76e21d\": container with ID starting with 2cd53ca63ba376ef66f3527ff980638c5b7c45418cdcfdec57d48e1d1d76e21d not found: ID does not exist" containerID="2cd53ca63ba376ef66f3527ff980638c5b7c45418cdcfdec57d48e1d1d76e21d"
Sep 18 18:58:44.423299 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.423133    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd53ca63ba376ef66f3527ff980638c5b7c45418cdcfdec57d48e1d1d76e21d"} err="failed to get container status \"2cd53ca63ba376ef66f3527ff980638c5b7c45418cdcfdec57d48e1d1d76e21d\": rpc error: code = NotFound desc = could not find container \"2cd53ca63ba376ef66f3527ff980638c5b7c45418cdcfdec57d48e1d1d76e21d\": container with ID starting with 2cd53ca63ba376ef66f3527ff980638c5b7c45418cdcfdec57d48e1d1d76e21d not found: ID does not exist"
Sep 18 18:58:44.425067 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.425045    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"]
Sep 18 18:58:44.429073 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.429049    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-62de055a-openshift-default-66b8c87c88-qk9jl"]
Sep 18 18:58:44.575868 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:44.575835    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f30ea4f-c89e-417a-bd84-c2e807855e37" path="/var/lib/kubelet/pods/1f30ea4f-c89e-417a-bd84-c2e807855e37/volumes"
Sep 18 18:58:45.921644 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:45.921606    2558 patch_prober.go:28] interesting pod/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 18:58:45.922032 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:45.921672    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch" podUID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 18:58:46.400865 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.400840    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:58:46.419175 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.419148    2558 generic.go:356] "Generic (PLEG): container finished" podID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerID="99a40c27ebeab44776884112a4fd42c98949273c227483d0e73919b34933d30c" exitCode=0
Sep 18 18:58:46.419330 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.419217    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"
Sep 18 18:58:46.419330 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.419216    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch" event={"ID":"f6d074b8-f8ae-4e64-b200-ad871d0d9473","Type":"ContainerDied","Data":"99a40c27ebeab44776884112a4fd42c98949273c227483d0e73919b34933d30c"}
Sep 18 18:58:46.419330 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.419269    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch" event={"ID":"f6d074b8-f8ae-4e64-b200-ad871d0d9473","Type":"ContainerDied","Data":"f650a3c84629f96defd1b1d5c077ab37dce222c20f0db66b5e6252a5cd7a4518"}
Sep 18 18:58:46.419330 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.419291    2558 scope.go:117] "RemoveContainer" containerID="99a40c27ebeab44776884112a4fd42c98949273c227483d0e73919b34933d30c"
Sep 18 18:58:46.430304 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.430284    2558 scope.go:117] "RemoveContainer" containerID="999c7a4d40436df6a06419a2cb28f046ed510269652f1fcf356c72b130765cf8"
Sep 18 18:58:46.439551 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.439532    2558 scope.go:117] "RemoveContainer" containerID="99a40c27ebeab44776884112a4fd42c98949273c227483d0e73919b34933d30c"
Sep 18 18:58:46.439821 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:58:46.439801    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a40c27ebeab44776884112a4fd42c98949273c227483d0e73919b34933d30c\": container with ID starting with 99a40c27ebeab44776884112a4fd42c98949273c227483d0e73919b34933d30c not found: ID does not exist" containerID="99a40c27ebeab44776884112a4fd42c98949273c227483d0e73919b34933d30c"
Sep 18 18:58:46.439882 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.439831    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a40c27ebeab44776884112a4fd42c98949273c227483d0e73919b34933d30c"} err="failed to get container status \"99a40c27ebeab44776884112a4fd42c98949273c227483d0e73919b34933d30c\": rpc error: code = NotFound desc = could not find container \"99a40c27ebeab44776884112a4fd42c98949273c227483d0e73919b34933d30c\": container with ID starting with 99a40c27ebeab44776884112a4fd42c98949273c227483d0e73919b34933d30c not found: ID does not exist"
Sep 18 18:58:46.439882 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.439849    2558 scope.go:117] "RemoveContainer" containerID="999c7a4d40436df6a06419a2cb28f046ed510269652f1fcf356c72b130765cf8"
Sep 18 18:58:46.440127 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:58:46.440109    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999c7a4d40436df6a06419a2cb28f046ed510269652f1fcf356c72b130765cf8\": container with ID starting with 999c7a4d40436df6a06419a2cb28f046ed510269652f1fcf356c72b130765cf8 not found: ID does not exist" containerID="999c7a4d40436df6a06419a2cb28f046ed510269652f1fcf356c72b130765cf8"
Sep 18 18:58:46.440215 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.440142    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999c7a4d40436df6a06419a2cb28f046ed510269652f1fcf356c72b130765cf8"} err="failed to get container status \"999c7a4d40436df6a06419a2cb28f046ed510269652f1fcf356c72b130765cf8\": rpc error: code = NotFound desc = could not find container \"999c7a4d40436df6a06419a2cb28f046ed510269652f1fcf356c72b130765cf8\": container with ID starting with 999c7a4d40436df6a06419a2cb28f046ed510269652f1fcf356c72b130765cf8 not found: ID does not exist"
Sep 18 18:58:46.473318 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.473292    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-workload-certs\") pod \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") "
Sep 18 18:58:46.473424 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.473329    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-workload-socket\") pod \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") "
Sep 18 18:58:46.473424 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.473362    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-podinfo\") pod \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") "
Sep 18 18:58:46.473424 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.473378    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-token\") pod \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") "
Sep 18 18:58:46.473424 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.473422    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-envoy\") pod \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") "
Sep 18 18:58:46.473678 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.473452    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istiod-ca-cert\") pod \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") "
Sep 18 18:58:46.473678 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.473485    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh8c9\" (UniqueName: \"kubernetes.io/projected/f6d074b8-f8ae-4e64-b200-ad871d0d9473-kube-api-access-jh8c9\") pod \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") "
Sep 18 18:58:46.473678 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.473522    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-credential-socket\") pod \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") "
Sep 18 18:58:46.473678 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.473549    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-data\") pod \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\" (UID: \"f6d074b8-f8ae-4e64-b200-ad871d0d9473\") "
Sep 18 18:58:46.473678 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.473598    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f6d074b8-f8ae-4e64-b200-ad871d0d9473" (UID: "f6d074b8-f8ae-4e64-b200-ad871d0d9473"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:58:46.473678 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.473646    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f6d074b8-f8ae-4e64-b200-ad871d0d9473" (UID: "f6d074b8-f8ae-4e64-b200-ad871d0d9473"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:58:46.474006 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.473901    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-data" (OuterVolumeSpecName: "istio-data") pod "f6d074b8-f8ae-4e64-b200-ad871d0d9473" (UID: "f6d074b8-f8ae-4e64-b200-ad871d0d9473"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:58:46.474006 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.473924    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f6d074b8-f8ae-4e64-b200-ad871d0d9473" (UID: "f6d074b8-f8ae-4e64-b200-ad871d0d9473"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:58:46.474006 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.473963    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:46.474006 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.473982    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:46.474232 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.474008    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f6d074b8-f8ae-4e64-b200-ad871d0d9473" (UID: "f6d074b8-f8ae-4e64-b200-ad871d0d9473"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:58:46.475644 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.475623    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f6d074b8-f8ae-4e64-b200-ad871d0d9473" (UID: "f6d074b8-f8ae-4e64-b200-ad871d0d9473"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:58:46.475755 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.475736    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f6d074b8-f8ae-4e64-b200-ad871d0d9473" (UID: "f6d074b8-f8ae-4e64-b200-ad871d0d9473"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 18:58:46.475807 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.475782    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-token" (OuterVolumeSpecName: "istio-token") pod "f6d074b8-f8ae-4e64-b200-ad871d0d9473" (UID: "f6d074b8-f8ae-4e64-b200-ad871d0d9473"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:58:46.475847 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.475813    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d074b8-f8ae-4e64-b200-ad871d0d9473-kube-api-access-jh8c9" (OuterVolumeSpecName: "kube-api-access-jh8c9") pod "f6d074b8-f8ae-4e64-b200-ad871d0d9473" (UID: "f6d074b8-f8ae-4e64-b200-ad871d0d9473"). InnerVolumeSpecName "kube-api-access-jh8c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:58:46.574868 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.574789    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:46.574868 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.574818    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:46.574868 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.574827    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:46.574868 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.574834    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:46.574868 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.574843    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh8c9\" (UniqueName: \"kubernetes.io/projected/f6d074b8-f8ae-4e64-b200-ad871d0d9473-kube-api-access-jh8c9\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:46.574868 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.574851    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:46.574868 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.574859    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6d074b8-f8ae-4e64-b200-ad871d0d9473-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:58:46.737577 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.737546    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"]
Sep 18 18:58:46.742472 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:46.742440    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-897f61-openshift-default-757d7cbcd6-5qvch"]
Sep 18 18:58:48.576830 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:58:48.576796    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" path="/var/lib/kubelet/pods/f6d074b8-f8ae-4e64-b200-ad871d0d9473/volumes"
Sep 18 18:59:00.998391 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.998357    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"]
Sep 18 18:59:00.999003 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.998985    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74bd161d-4fb3-4a8c-8ecb-5437e55b4842" containerName="revoke-keys"
Sep 18 18:59:00.999055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999005    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="74bd161d-4fb3-4a8c-8ecb-5437e55b4842" containerName="revoke-keys"
Sep 18 18:59:00.999055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999021    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerName="istio-proxy"
Sep 18 18:59:00.999055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999030    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerName="istio-proxy"
Sep 18 18:59:00.999055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999040    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30bb7138-5200-4731-ab2b-637bfacd57eb" containerName="revoke-keys"
Sep 18 18:59:00.999055 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999049    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="30bb7138-5200-4731-ab2b-637bfacd57eb" containerName="revoke-keys"
Sep 18 18:59:00.999230 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999066    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f30ea4f-c89e-417a-bd84-c2e807855e37" containerName="istio-proxy"
Sep 18 18:59:00.999230 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999074    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f30ea4f-c89e-417a-bd84-c2e807855e37" containerName="istio-proxy"
Sep 18 18:59:00.999230 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999093    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerName="istio-proxy"
Sep 18 18:59:00.999230 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999101    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerName="istio-proxy"
Sep 18 18:59:00.999230 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999111    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f30ea4f-c89e-417a-bd84-c2e807855e37" containerName="istio-proxy"
Sep 18 18:59:00.999230 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999120    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f30ea4f-c89e-417a-bd84-c2e807855e37" containerName="istio-proxy"
Sep 18 18:59:00.999230 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999229    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerName="istio-proxy"
Sep 18 18:59:00.999454 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999240    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f30ea4f-c89e-417a-bd84-c2e807855e37" containerName="istio-proxy"
Sep 18 18:59:00.999454 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999252    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6d074b8-f8ae-4e64-b200-ad871d0d9473" containerName="istio-proxy"
Sep 18 18:59:00.999454 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999266    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="74bd161d-4fb3-4a8c-8ecb-5437e55b4842" containerName="revoke-keys"
Sep 18 18:59:00.999454 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:00.999277    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="30bb7138-5200-4731-ab2b-637bfacd57eb" containerName="revoke-keys"
Sep 18 18:59:01.003211 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.003178    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.007317 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.007296    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-a6d509-openshift-default-dockercfg-vg8cf\""
Sep 18 18:59:01.022559 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.022535    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-workload-socket\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.022667 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.022582    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-data\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.022667 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.022606    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-credential-socket\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.022792 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.022674    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-workload-certs\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.022792 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.022757    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-token\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.022899 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.022802    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de4ecae-a309-4b46-9562-2a532f0fb609-istiod-ca-cert\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.022899 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.022867    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-envoy\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.022974 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.022914    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-podinfo\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.022974 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.022941    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47fgr\" (UniqueName: \"kubernetes.io/projected/4de4ecae-a309-4b46-9562-2a532f0fb609-kube-api-access-47fgr\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.052576 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.052546    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"]
Sep 18 18:59:01.124029 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.123998    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-workload-socket\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.124248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.124054    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-data\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.124248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.124091    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-credential-socket\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.124248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.124115    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-workload-certs\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.124248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.124161    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-token\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.124248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.124215    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de4ecae-a309-4b46-9562-2a532f0fb609-istiod-ca-cert\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.124539 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.124263    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-envoy\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.124539 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.124303    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-podinfo\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.124539 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.124328    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47fgr\" (UniqueName: \"kubernetes.io/projected/4de4ecae-a309-4b46-9562-2a532f0fb609-kube-api-access-47fgr\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.124539 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.124492    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-workload-socket\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.124539 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.124521    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-data\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.124856 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.124556    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-workload-certs\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.124856 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.124590    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-credential-socket\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.125222 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.125157    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de4ecae-a309-4b46-9562-2a532f0fb609-istiod-ca-cert\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.126755 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.126720    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-envoy\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.126976 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.126961    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-podinfo\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.138233 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.138210    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-token\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.138646 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.138625    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47fgr\" (UniqueName: \"kubernetes.io/projected/4de4ecae-a309-4b46-9562-2a532f0fb609-kube-api-access-47fgr\") pod \"e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") " pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.313597 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.313504    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:01.606899 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.606868    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"]
Sep 18 18:59:01.607441 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:59:01.607410    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de4ecae_a309_4b46_9562_2a532f0fb609.slice/crio-55ff8f9d228a757fff4b2a16ae09386002893003b17d50b40f813adf51dcb283 WatchSource:0}: Error finding container 55ff8f9d228a757fff4b2a16ae09386002893003b17d50b40f813adf51dcb283: Status 404 returned error can't find the container with id 55ff8f9d228a757fff4b2a16ae09386002893003b17d50b40f813adf51dcb283
Sep 18 18:59:01.609746 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.609714    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:59:01.609846 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.609777    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:59:01.609846 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:01.609810    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:59:02.537077 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:02.537037    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29" event={"ID":"4de4ecae-a309-4b46-9562-2a532f0fb609","Type":"ContainerStarted","Data":"44918cfabe3bddf252503e1e822e57a1d70a728fef00d1801683ba84893965ff"}
Sep 18 18:59:02.537570 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:02.537132    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29" event={"ID":"4de4ecae-a309-4b46-9562-2a532f0fb609","Type":"ContainerStarted","Data":"55ff8f9d228a757fff4b2a16ae09386002893003b17d50b40f813adf51dcb283"}
Sep 18 18:59:02.603225 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:02.603138    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29" podStartSLOduration=2.603119087 podStartE2EDuration="2.603119087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:02.599630416 +0000 UTC m=+1202.036209259" watchObservedRunningTime="2026-09-18 18:59:02.603119087 +0000 UTC m=+1202.039697930"
Sep 18 18:59:02.657039 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:02.657012    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 18:59:02.657239 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:02.657018    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 18:59:02.660044 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:02.660022    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 18:59:02.660044 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:02.660035    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 18:59:03.314153 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:03.314119    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:03.319849 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:03.319825    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:03.543099 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:03.543063    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:03.543876 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:03.543860    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:23.426457 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.426421    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"]
Sep 18 18:59:23.427056 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.427032    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f30ea4f-c89e-417a-bd84-c2e807855e37" containerName="istio-proxy"
Sep 18 18:59:23.430277 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.430256    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.433794 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.433774    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-351045-openshift-default-dockercfg-ck9vz\""
Sep 18 18:59:23.466040 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.466009    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"]
Sep 18 18:59:23.556656 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.556611    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-envoy\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.556831 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.556664    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-data\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.556831 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.556728    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-token\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.556831 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.556762    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx66k\" (UniqueName: \"kubernetes.io/projected/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-kube-api-access-jx66k\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.556953 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.556835    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-credential-socket\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.556953 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.556859    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-workload-certs\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.556953 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.556885    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-workload-socket\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.557084 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.557052    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istiod-ca-cert\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.557124 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.557103    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-podinfo\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.658398 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.658365    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-envoy\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.658606 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.658408    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-data\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.658606 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.658429    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-token\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.658606 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.658445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx66k\" (UniqueName: \"kubernetes.io/projected/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-kube-api-access-jx66k\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.658606 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.658528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-credential-socket\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.658606 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.658560    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-workload-certs\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.658606 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.658586    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-workload-socket\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.658935 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.658690    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istiod-ca-cert\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.658935 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.658717    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-podinfo\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.658935 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.658912    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-data\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.659150 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.658974    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-workload-certs\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.659150 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.659127    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-workload-socket\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.659288 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.659205    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-credential-socket\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.659726 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.659701    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istiod-ca-cert\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.661102 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.661072    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-envoy\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.661382 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.661364    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-podinfo\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.676978 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.676922    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-token\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.677208 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.677162    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx66k\" (UniqueName: \"kubernetes.io/projected/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-kube-api-access-jx66k\") pod \"e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") " pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:23.741710 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:23.741672    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:24.022042 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:24.022019    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"]
Sep 18 18:59:24.023316 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:59:24.023289    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef653752_c2ed_41cc_89ed_ee218cdb1d8a.slice/crio-03ad71c448120740727c86dc6faa1278232cdb9b9668600c70450ea54b83c705 WatchSource:0}: Error finding container 03ad71c448120740727c86dc6faa1278232cdb9b9668600c70450ea54b83c705: Status 404 returned error can't find the container with id 03ad71c448120740727c86dc6faa1278232cdb9b9668600c70450ea54b83c705
Sep 18 18:59:24.025718 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:24.025687    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:59:24.025771 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:24.025750    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:59:24.025810 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:24.025782    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:59:24.682553 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:24.682518    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4" event={"ID":"ef653752-c2ed-41cc-89ed-ee218cdb1d8a","Type":"ContainerStarted","Data":"381a3598e5cbe4238939274e6fb5eacc70fa94591877cfb112d01dccb6b54205"}
Sep 18 18:59:24.682945 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:24.682582    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4" event={"ID":"ef653752-c2ed-41cc-89ed-ee218cdb1d8a","Type":"ContainerStarted","Data":"03ad71c448120740727c86dc6faa1278232cdb9b9668600c70450ea54b83c705"}
Sep 18 18:59:24.707608 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:24.707557    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4" podStartSLOduration=1.70754142 podStartE2EDuration="1.70754142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:24.70613856 +0000 UTC m=+1224.142717402" watchObservedRunningTime="2026-09-18 18:59:24.70754142 +0000 UTC m=+1224.144120262"
Sep 18 18:59:24.742571 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:24.742539    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:25.747180 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:25.747147    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:26.697803 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:26.697763    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:26.699003 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:26.698977    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:34.585367 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:34.585327    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 18 18:59:34.585909 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:34.585404    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29" podUID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 18:59:36.779232 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:36.779170    2558 generic.go:356] "Generic (PLEG): container finished" podID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerID="44918cfabe3bddf252503e1e822e57a1d70a728fef00d1801683ba84893965ff" exitCode=0
Sep 18 18:59:36.779779 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:36.779227    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29" event={"ID":"4de4ecae-a309-4b46-9562-2a532f0fb609","Type":"ContainerDied","Data":"44918cfabe3bddf252503e1e822e57a1d70a728fef00d1801683ba84893965ff"}
Sep 18 18:59:36.779857 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:36.779794    2558 scope.go:117] "RemoveContainer" containerID="44918cfabe3bddf252503e1e822e57a1d70a728fef00d1801683ba84893965ff"
Sep 18 18:59:36.781051 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:36.781018    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:59:36.781141 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:36.781089    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:59:36.781141 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:36.781123    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:59:37.789452 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:37.789417    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29" event={"ID":"4de4ecae-a309-4b46-9562-2a532f0fb609","Type":"ContainerStarted","Data":"0ca9ff965c88c1c96086aec731a04d53fb197f85f39c86a8b0e3ceaf93019f26"}
Sep 18 18:59:38.315070 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:38.315030    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:38.315350 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:38.315321    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 18 18:59:38.315440 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:38.315417    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29" podUID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 18 18:59:39.314821 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:39.314779    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 18 18:59:39.315326 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:39.314858    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29" podUID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 18 18:59:40.314598 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:40.314564    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 18 18:59:40.314761 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:40.314620    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29" podUID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 18 18:59:41.314627 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:41.314584    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 18 18:59:41.315036 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:41.314686    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29" podUID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 18 18:59:42.314005 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:42.313971    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Sep 18 18:59:42.314168 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:42.314028    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29" podUID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Sep 18 18:59:43.317504 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:43.317471    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:43.317877 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:43.317761    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:43.318384 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:43.318364    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 18:59:52.397702 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:52.397616    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8950-3e7a5a"]
Sep 18 18:59:52.401422 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:52.401393    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-3e7a5a"
Sep 18 18:59:52.404510 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:52.404485    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7m4wl\""
Sep 18 18:59:52.406172 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:52.406147    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8950-3e7a5a"]
Sep 18 18:59:52.467209 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:52.466807    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h756\" (UniqueName: \"kubernetes.io/projected/4f277c40-dc96-4a4b-acb1-6e81bb060f96-kube-api-access-9h756\") pod \"test-curl-8950-3e7a5a\" (UID: \"4f277c40-dc96-4a4b-acb1-6e81bb060f96\") " pod="opendatahub/test-curl-8950-3e7a5a"
Sep 18 18:59:52.568378 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:52.568340    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h756\" (UniqueName: \"kubernetes.io/projected/4f277c40-dc96-4a4b-acb1-6e81bb060f96-kube-api-access-9h756\") pod \"test-curl-8950-3e7a5a\" (UID: \"4f277c40-dc96-4a4b-acb1-6e81bb060f96\") " pod="opendatahub/test-curl-8950-3e7a5a"
Sep 18 18:59:52.578296 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:52.578263    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h756\" (UniqueName: \"kubernetes.io/projected/4f277c40-dc96-4a4b-acb1-6e81bb060f96-kube-api-access-9h756\") pod \"test-curl-8950-3e7a5a\" (UID: \"4f277c40-dc96-4a4b-acb1-6e81bb060f96\") " pod="opendatahub/test-curl-8950-3e7a5a"
Sep 18 18:59:52.712589 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:52.712508    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-3e7a5a"
Sep 18 18:59:53.015752 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:53.015705    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8950-3e7a5a"]
Sep 18 18:59:53.017927 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:59:53.017901    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f277c40_dc96_4a4b_acb1_6e81bb060f96.slice/crio-b774e28a49588935c1699d1e6a2fa3d5100ed9db2d2449084891aae9eb5ea93c WatchSource:0}: Error finding container b774e28a49588935c1699d1e6a2fa3d5100ed9db2d2449084891aae9eb5ea93c: Status 404 returned error can't find the container with id b774e28a49588935c1699d1e6a2fa3d5100ed9db2d2449084891aae9eb5ea93c
Sep 18 18:59:53.905934 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:53.905902    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8950-3e7a5a" event={"ID":"4f277c40-dc96-4a4b-acb1-6e81bb060f96","Type":"ContainerStarted","Data":"667ecb6f2862547916dd0e92230d62594850e6e61527a0d4dc8c8d16ebcb917e"}
Sep 18 18:59:53.906469 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:53.905946    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8950-3e7a5a" event={"ID":"4f277c40-dc96-4a4b-acb1-6e81bb060f96","Type":"ContainerStarted","Data":"b774e28a49588935c1699d1e6a2fa3d5100ed9db2d2449084891aae9eb5ea93c"}
Sep 18 18:59:53.923912 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:53.923856    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8950-3e7a5a" podStartSLOduration=1.9238402319999999 podStartE2EDuration="1.923840232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:53.922295757 +0000 UTC m=+1253.358874602" watchObservedRunningTime="2026-09-18 18:59:53.923840232 +0000 UTC m=+1253.360419075"
Sep 18 18:59:54.545059 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:54.545020    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8950-3e7a5a"]
Sep 18 18:59:54.549549 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:54.549501    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8950-3e7a5a"]
Sep 18 18:59:54.654315 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:54.654274    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8950-1be2d2"]
Sep 18 18:59:54.655015 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:54.654994    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f277c40-dc96-4a4b-acb1-6e81bb060f96" containerName="test-curl-8950-3e7a5a"
Sep 18 18:59:54.655065 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:54.655019    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f277c40-dc96-4a4b-acb1-6e81bb060f96" containerName="test-curl-8950-3e7a5a"
Sep 18 18:59:54.655149 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:54.655137    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f277c40-dc96-4a4b-acb1-6e81bb060f96" containerName="test-curl-8950-3e7a5a"
Sep 18 18:59:54.658987 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:54.658968    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-1be2d2"
Sep 18 18:59:54.664549 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:54.664524    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8950-1be2d2"]
Sep 18 18:59:54.796585 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:54.796547    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l6kr\" (UniqueName: \"kubernetes.io/projected/17ce291a-62ba-47d6-a4e7-1b4c3ed05980-kube-api-access-6l6kr\") pod \"test-curl-8950-1be2d2\" (UID: \"17ce291a-62ba-47d6-a4e7-1b4c3ed05980\") " pod="opendatahub/test-curl-8950-1be2d2"
Sep 18 18:59:54.898334 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:54.898293    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l6kr\" (UniqueName: \"kubernetes.io/projected/17ce291a-62ba-47d6-a4e7-1b4c3ed05980-kube-api-access-6l6kr\") pod \"test-curl-8950-1be2d2\" (UID: \"17ce291a-62ba-47d6-a4e7-1b4c3ed05980\") " pod="opendatahub/test-curl-8950-1be2d2"
Sep 18 18:59:54.913012 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:54.910747    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l6kr\" (UniqueName: \"kubernetes.io/projected/17ce291a-62ba-47d6-a4e7-1b4c3ed05980-kube-api-access-6l6kr\") pod \"test-curl-8950-1be2d2\" (UID: \"17ce291a-62ba-47d6-a4e7-1b4c3ed05980\") " pod="opendatahub/test-curl-8950-1be2d2"
Sep 18 18:59:54.917078 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:59:54.917047    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8950-3e7a5a" container="test-curl-8950-3e7a5a"
Sep 18 18:59:54.972996 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:54.972956    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-1be2d2"
Sep 18 18:59:55.251291 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:55.251262    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8950-1be2d2"]
Sep 18 18:59:55.254220 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:59:55.254169    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17ce291a_62ba_47d6_a4e7_1b4c3ed05980.slice/crio-e7686c8bf3ad39d80671f78bff8421b41eaefd5e7389596aa71f4f1278435402 WatchSource:0}: Error finding container e7686c8bf3ad39d80671f78bff8421b41eaefd5e7389596aa71f4f1278435402: Status 404 returned error can't find the container with id e7686c8bf3ad39d80671f78bff8421b41eaefd5e7389596aa71f4f1278435402
Sep 18 18:59:55.926098 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:55.926062    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8950-1be2d2" event={"ID":"17ce291a-62ba-47d6-a4e7-1b4c3ed05980","Type":"ContainerStarted","Data":"b7a4572e1cdfb3186b86eec8fa783ebea93b198b10fd4ee701c59aa3c2b6e121"}
Sep 18 18:59:55.926098 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:55.926108    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8950-1be2d2" event={"ID":"17ce291a-62ba-47d6-a4e7-1b4c3ed05980","Type":"ContainerStarted","Data":"e7686c8bf3ad39d80671f78bff8421b41eaefd5e7389596aa71f4f1278435402"}
Sep 18 18:59:55.926650 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:55.926287    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8950-3e7a5a" podUID="4f277c40-dc96-4a4b-acb1-6e81bb060f96" containerName="test-curl-8950-3e7a5a" containerID="cri-o://667ecb6f2862547916dd0e92230d62594850e6e61527a0d4dc8c8d16ebcb917e" gracePeriod=2
Sep 18 18:59:55.944042 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:55.943998    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8950-1be2d2" podStartSLOduration=1.9439839540000001 podStartE2EDuration="1.943983954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:55.941274567 +0000 UTC m=+1255.377853409" watchObservedRunningTime="2026-09-18 18:59:55.943983954 +0000 UTC m=+1255.380562796"
Sep 18 18:59:56.551179 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.551134    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8950-1be2d2"]
Sep 18 18:59:56.555726 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.555698    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8950-1be2d2"]
Sep 18 18:59:56.673003 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.672972    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8950-c50cb7"]
Sep 18 18:59:56.673480 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.673465    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17ce291a-62ba-47d6-a4e7-1b4c3ed05980" containerName="test-curl-8950-1be2d2"
Sep 18 18:59:56.673525 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.673483    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="17ce291a-62ba-47d6-a4e7-1b4c3ed05980" containerName="test-curl-8950-1be2d2"
Sep 18 18:59:56.673593 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.673582    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="17ce291a-62ba-47d6-a4e7-1b4c3ed05980" containerName="test-curl-8950-1be2d2"
Sep 18 18:59:56.676881 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.676865    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-c50cb7"
Sep 18 18:59:56.683138 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.683113    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8950-c50cb7"]
Sep 18 18:59:56.818398 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.818362    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgvdx\" (UniqueName: \"kubernetes.io/projected/3b223294-abd5-41ff-af65-7dce203efdc2-kube-api-access-pgvdx\") pod \"test-curl-8950-c50cb7\" (UID: \"3b223294-abd5-41ff-af65-7dce203efdc2\") " pod="opendatahub/test-curl-8950-c50cb7"
Sep 18 18:59:56.861910 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.861874    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": read tcp 10.132.0.2:39414->10.132.0.80:15021: read: connection reset by peer" start-of-body=
Sep 18 18:59:56.862051 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.861946    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4" podUID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": read tcp 10.132.0.2:39414->10.132.0.80:15021: read: connection reset by peer"
Sep 18 18:59:56.919345 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.919308    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgvdx\" (UniqueName: \"kubernetes.io/projected/3b223294-abd5-41ff-af65-7dce203efdc2-kube-api-access-pgvdx\") pod \"test-curl-8950-c50cb7\" (UID: \"3b223294-abd5-41ff-af65-7dce203efdc2\") " pod="opendatahub/test-curl-8950-c50cb7"
Sep 18 18:59:56.928239 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.928164    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgvdx\" (UniqueName: \"kubernetes.io/projected/3b223294-abd5-41ff-af65-7dce203efdc2-kube-api-access-pgvdx\") pod \"test-curl-8950-c50cb7\" (UID: \"3b223294-abd5-41ff-af65-7dce203efdc2\") " pod="opendatahub/test-curl-8950-c50cb7"
Sep 18 18:59:56.933572 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.933547    2558 generic.go:356] "Generic (PLEG): container finished" podID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerID="381a3598e5cbe4238939274e6fb5eacc70fa94591877cfb112d01dccb6b54205" exitCode=0
Sep 18 18:59:56.933683 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.933615    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4" event={"ID":"ef653752-c2ed-41cc-89ed-ee218cdb1d8a","Type":"ContainerDied","Data":"381a3598e5cbe4238939274e6fb5eacc70fa94591877cfb112d01dccb6b54205"}
Sep 18 18:59:56.934008 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:59:56.933989    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8950-1be2d2" container="test-curl-8950-1be2d2"
Sep 18 18:59:56.934008 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.934000    2558 scope.go:117] "RemoveContainer" containerID="381a3598e5cbe4238939274e6fb5eacc70fa94591877cfb112d01dccb6b54205"
Sep 18 18:59:56.935272 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.935236    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:59:56.935351 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.935326    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:59:56.935391 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.935374    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 18:59:56.989159 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:56.989134    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-c50cb7"
Sep 18 18:59:57.281646 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:57.281612    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8950-c50cb7"]
Sep 18 18:59:57.282954 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:59:57.282926    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b223294_abd5_41ff_af65_7dce203efdc2.slice/crio-a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764 WatchSource:0}: Error finding container a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764: Status 404 returned error can't find the container with id a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764
Sep 18 18:59:57.941851 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:57.941818    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8950-c50cb7" event={"ID":"3b223294-abd5-41ff-af65-7dce203efdc2","Type":"ContainerStarted","Data":"a4eb4bf3a96105072e9f2c46e2fdb3d7d0d62635520836f20391a8d90086a5de"}
Sep 18 18:59:57.942248 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:57.941861    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8950-c50cb7" event={"ID":"3b223294-abd5-41ff-af65-7dce203efdc2","Type":"ContainerStarted","Data":"a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764"}
Sep 18 18:59:57.944243 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:57.944220    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4" event={"ID":"ef653752-c2ed-41cc-89ed-ee218cdb1d8a","Type":"ContainerStarted","Data":"4673b9343bbd3626c0c16417275c4ae77c6162a2e33080ef0353c82a5a57e90f"}
Sep 18 18:59:57.944360 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:57.944336    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8950-1be2d2" podUID="17ce291a-62ba-47d6-a4e7-1b4c3ed05980" containerName="test-curl-8950-1be2d2" containerID="cri-o://b7a4572e1cdfb3186b86eec8fa783ebea93b198b10fd4ee701c59aa3c2b6e121" gracePeriod=2
Sep 18 18:59:57.960268 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:57.958380    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8950-c50cb7" podStartSLOduration=1.958355728 podStartE2EDuration="1.958355728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:57.955357219 +0000 UTC m=+1257.391936063" watchObservedRunningTime="2026-09-18 18:59:57.958355728 +0000 UTC m=+1257.394934573"
Sep 18 18:59:58.182142 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.182048    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-3e7a5a"
Sep 18 18:59:58.337174 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.337140    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h756\" (UniqueName: \"kubernetes.io/projected/4f277c40-dc96-4a4b-acb1-6e81bb060f96-kube-api-access-9h756\") pod \"4f277c40-dc96-4a4b-acb1-6e81bb060f96\" (UID: \"4f277c40-dc96-4a4b-acb1-6e81bb060f96\") "
Sep 18 18:59:58.339478 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.339442    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f277c40-dc96-4a4b-acb1-6e81bb060f96-kube-api-access-9h756" (OuterVolumeSpecName: "kube-api-access-9h756") pod "4f277c40-dc96-4a4b-acb1-6e81bb060f96" (UID: "4f277c40-dc96-4a4b-acb1-6e81bb060f96"). InnerVolumeSpecName "kube-api-access-9h756". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:59:58.438734 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.438653    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h756\" (UniqueName: \"kubernetes.io/projected/4f277c40-dc96-4a4b-acb1-6e81bb060f96-kube-api-access-9h756\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 18:59:58.566143 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.565576    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8950-c50cb7"]
Sep 18 18:59:58.583351 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.583318    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f277c40-dc96-4a4b-acb1-6e81bb060f96" path="/var/lib/kubelet/pods/4f277c40-dc96-4a4b-acb1-6e81bb060f96/volumes"
Sep 18 18:59:58.583739 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.583723    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8950-c50cb7"]
Sep 18 18:59:58.677274 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.677234    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8950-4aa2b2"]
Sep 18 18:59:58.677697 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.677682    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b223294-abd5-41ff-af65-7dce203efdc2" containerName="test-curl-8950-c50cb7"
Sep 18 18:59:58.677745 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.677699    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b223294-abd5-41ff-af65-7dce203efdc2" containerName="test-curl-8950-c50cb7"
Sep 18 18:59:58.677797 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.677788    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b223294-abd5-41ff-af65-7dce203efdc2" containerName="test-curl-8950-c50cb7"
Sep 18 18:59:58.681131 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.681116    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-4aa2b2"
Sep 18 18:59:58.686793 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.686770    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8950-4aa2b2"]
Sep 18 18:59:58.741820 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.741790    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 18:59:58.742040 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.742018    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 18 18:59:58.742135 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.742074    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4" podUID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 18 18:59:58.842898 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.842860    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wsgr\" (UniqueName: \"kubernetes.io/projected/ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6-kube-api-access-6wsgr\") pod \"test-curl-8950-4aa2b2\" (UID: \"ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6\") " pod="opendatahub/test-curl-8950-4aa2b2"
Sep 18 18:59:58.944433 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.944332    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wsgr\" (UniqueName: \"kubernetes.io/projected/ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6-kube-api-access-6wsgr\") pod \"test-curl-8950-4aa2b2\" (UID: \"ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6\") " pod="opendatahub/test-curl-8950-4aa2b2"
Sep 18 18:59:58.954467 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.954431    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wsgr\" (UniqueName: \"kubernetes.io/projected/ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6-kube-api-access-6wsgr\") pod \"test-curl-8950-4aa2b2\" (UID: \"ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6\") " pod="opendatahub/test-curl-8950-4aa2b2"
Sep 18 18:59:58.958544 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.958505    2558 generic.go:356] "Generic (PLEG): container finished" podID="4f277c40-dc96-4a4b-acb1-6e81bb060f96" containerID="667ecb6f2862547916dd0e92230d62594850e6e61527a0d4dc8c8d16ebcb917e" exitCode=137
Sep 18 18:59:58.958706 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.958590    2558 scope.go:117] "RemoveContainer" containerID="667ecb6f2862547916dd0e92230d62594850e6e61527a0d4dc8c8d16ebcb917e"
Sep 18 18:59:58.959095 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.959075    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-3e7a5a"
Sep 18 18:59:58.959646 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:59:58.959623    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8950-c50cb7" container="test-curl-8950-c50cb7"
Sep 18 18:59:58.985099 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.985076    2558 scope.go:117] "RemoveContainer" containerID="667ecb6f2862547916dd0e92230d62594850e6e61527a0d4dc8c8d16ebcb917e"
Sep 18 18:59:58.985467 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:59:58.985441    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"667ecb6f2862547916dd0e92230d62594850e6e61527a0d4dc8c8d16ebcb917e\": container with ID starting with 667ecb6f2862547916dd0e92230d62594850e6e61527a0d4dc8c8d16ebcb917e not found: ID does not exist" containerID="667ecb6f2862547916dd0e92230d62594850e6e61527a0d4dc8c8d16ebcb917e"
Sep 18 18:59:58.985556 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.985482    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667ecb6f2862547916dd0e92230d62594850e6e61527a0d4dc8c8d16ebcb917e"} err="failed to get container status \"667ecb6f2862547916dd0e92230d62594850e6e61527a0d4dc8c8d16ebcb917e\": rpc error: code = NotFound desc = could not find container \"667ecb6f2862547916dd0e92230d62594850e6e61527a0d4dc8c8d16ebcb917e\": container with ID starting with 667ecb6f2862547916dd0e92230d62594850e6e61527a0d4dc8c8d16ebcb917e not found: ID does not exist"
Sep 18 18:59:58.992602 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:58.992586    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-4aa2b2"
Sep 18 18:59:59.272681 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:59.272060    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8950-4aa2b2"]
Sep 18 18:59:59.276737 ip-10-0-130-78 kubenswrapper[2558]: W0918 18:59:59.276348    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podede3f37c_8d4a_4a70_8a59_d5e2b7e65bb6.slice/crio-9a13c21c4e8db8d34b2ce270f150a7a3769d3888ce1625b910224f9160950468 WatchSource:0}: Error finding container 9a13c21c4e8db8d34b2ce270f150a7a3769d3888ce1625b910224f9160950468: Status 404 returned error can't find the container with id 9a13c21c4e8db8d34b2ce270f150a7a3769d3888ce1625b910224f9160950468
Sep 18 18:59:59.742381 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:59.742350    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 18 18:59:59.742553 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:59.742404    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4" podUID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 18 18:59:59.977311 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:59.975209    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8950-4aa2b2" event={"ID":"ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6","Type":"ContainerStarted","Data":"004ce2a588ec4cace5f4169c3f6ca01d7c1476c463605579818493016c52f47d"}
Sep 18 18:59:59.977311 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:59.975266    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8950-4aa2b2" event={"ID":"ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6","Type":"ContainerStarted","Data":"9a13c21c4e8db8d34b2ce270f150a7a3769d3888ce1625b910224f9160950468"}
Sep 18 18:59:59.980432 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:59.980402    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8950-c50cb7" podUID="3b223294-abd5-41ff-af65-7dce203efdc2" containerName="test-curl-8950-c50cb7" containerID="cri-o://a4eb4bf3a96105072e9f2c46e2fdb3d7d0d62635520836f20391a8d90086a5de" gracePeriod=2
Sep 18 18:59:59.992423 ip-10-0-130-78 kubenswrapper[2558]: I0918 18:59:59.992343    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8950-4aa2b2" podStartSLOduration=1.992327137 podStartE2EDuration="1.992327137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:59.990628771 +0000 UTC m=+1259.427207614" watchObservedRunningTime="2026-09-18 18:59:59.992327137 +0000 UTC m=+1259.428905978"
Sep 18 18:59:59.992618 ip-10-0-130-78 kubenswrapper[2558]: E0918 18:59:59.992591    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8950-c50cb7\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="3b223294-abd5-41ff-af65-7dce203efdc2" pod="opendatahub/test-curl-8950-c50cb7"
Sep 18 19:00:00.173207 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.172381    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-880bee-29829300-d665s"]
Sep 18 19:00:00.184002 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.180762    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-298125-29829300-ctd75"]
Sep 18 19:00:00.184002 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.181232    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-880bee-29829300-d665s"
Sep 18 19:00:00.184726 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.184150    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 19:00:00.200721 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.187651    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829300-h6rcs"]
Sep 18 19:00:00.200721 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.190825    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-298125-29829300-ctd75"
Sep 18 19:00:00.205657 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.205627    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-1be2d2"
Sep 18 19:00:00.208426 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.206358    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9d9f36-29829300-fhd76"]
Sep 18 19:00:00.208426 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.206532    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-h6rcs"
Sep 18 19:00:00.208426 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:00.207432    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8950-c50cb7\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="3b223294-abd5-41ff-af65-7dce203efdc2" pod="opendatahub/test-curl-8950-c50cb7"
Sep 18 19:00:00.209781 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:00.209738    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8950-c50cb7\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="3b223294-abd5-41ff-af65-7dce203efdc2" pod="opendatahub/test-curl-8950-c50cb7"
Sep 18 19:00:00.212282 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.210954    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-880bee-29829300-d665s"]
Sep 18 19:00:00.212282 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.211017    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829300-h6rcs"]
Sep 18 19:00:00.212282 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.211123    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9d9f36-29829300-fhd76"
Sep 18 19:00:00.214123 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.214100    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9d9f36-29829300-fhd76"]
Sep 18 19:00:00.218673 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.218655    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-880bee-29829300-d665s"
Sep 18 19:00:00.219440 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.219425    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-298125-29829300-ctd75"]
Sep 18 19:00:00.230616 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.229990    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cc289c-29829300-k8sw7"]
Sep 18 19:00:00.230616 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.230119    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-298125-29829300-ctd75"
Sep 18 19:00:00.234636 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.234445    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cc289c-29829300-k8sw7"]
Sep 18 19:00:00.234636 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.234546    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cc289c-29829300-k8sw7"
Sep 18 19:00:00.242499 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:00.242434    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8950-c50cb7\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="3b223294-abd5-41ff-af65-7dce203efdc2" pod="opendatahub/test-curl-8950-c50cb7"
Sep 18 19:00:00.284765 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.284411    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-h6rcs"
Sep 18 19:00:00.297307 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.297282    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9d9f36-29829300-fhd76"
Sep 18 19:00:00.304450 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.304431    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cc289c-29829300-k8sw7"
Sep 18 19:00:00.387361 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.383589    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l6kr\" (UniqueName: \"kubernetes.io/projected/17ce291a-62ba-47d6-a4e7-1b4c3ed05980-kube-api-access-6l6kr\") pod \"17ce291a-62ba-47d6-a4e7-1b4c3ed05980\" (UID: \"17ce291a-62ba-47d6-a4e7-1b4c3ed05980\") "
Sep 18 19:00:00.392296 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.392151    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ce291a-62ba-47d6-a4e7-1b4c3ed05980-kube-api-access-6l6kr" (OuterVolumeSpecName: "kube-api-access-6l6kr") pod "17ce291a-62ba-47d6-a4e7-1b4c3ed05980" (UID: "17ce291a-62ba-47d6-a4e7-1b4c3ed05980"). InnerVolumeSpecName "kube-api-access-6l6kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:00:00.485453 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.485417    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l6kr\" (UniqueName: \"kubernetes.io/projected/17ce291a-62ba-47d6-a4e7-1b4c3ed05980-kube-api-access-6l6kr\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:00.553229 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.546354    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-880bee-29829300-d665s"]
Sep 18 19:00:00.553229 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:00:00.550429    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444696b0_826c_4fe2_b096_52ed6c0ade7d.slice/crio-1e65b788eab89d5131087d587467c3d19acf5a58d025a8d4fc21779cac41961d WatchSource:0}: Error finding container 1e65b788eab89d5131087d587467c3d19acf5a58d025a8d4fc21779cac41961d: Status 404 returned error can't find the container with id 1e65b788eab89d5131087d587467c3d19acf5a58d025a8d4fc21779cac41961d
Sep 18 19:00:00.587297 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.587262    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17ce291a-62ba-47d6-a4e7-1b4c3ed05980" path="/var/lib/kubelet/pods/17ce291a-62ba-47d6-a4e7-1b4c3ed05980/volumes"
Sep 18 19:00:00.587744 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.587723    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-298125-29829300-ctd75"]
Sep 18 19:00:00.589097 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:00:00.589073    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88696862_bcce_4ac3_b840_a541420b28a6.slice/crio-c001efb8a66747cbab44746daa26c23fdf2ae14a22fa742d0f81ac586c9118bc WatchSource:0}: Error finding container c001efb8a66747cbab44746daa26c23fdf2ae14a22fa742d0f81ac586c9118bc: Status 404 returned error can't find the container with id c001efb8a66747cbab44746daa26c23fdf2ae14a22fa742d0f81ac586c9118bc
Sep 18 19:00:00.634672 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:00:00.634644    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d731b4f_9164_408f_a14d_229567170101.slice/crio-dac047e688aa91bb1aefbf1c9926548881b5002c72f7f2c558014604bb9c6bbc WatchSource:0}: Error finding container dac047e688aa91bb1aefbf1c9926548881b5002c72f7f2c558014604bb9c6bbc: Status 404 returned error can't find the container with id dac047e688aa91bb1aefbf1c9926548881b5002c72f7f2c558014604bb9c6bbc
Sep 18 19:00:00.634852 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.634813    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829300-h6rcs"]
Sep 18 19:00:00.675205 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.673258    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9d9f36-29829300-fhd76"]
Sep 18 19:00:00.675205 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.674676    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cc289c-29829300-k8sw7"]
Sep 18 19:00:00.684670 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.684647    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8950-4aa2b2"]
Sep 18 19:00:00.689213 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.688935    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8950-4aa2b2"]
Sep 18 19:00:00.742732 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.742702    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 18 19:00:00.742878 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.742756    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4" podUID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 18 19:00:00.989772 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.989623    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-h6rcs" event={"ID":"9d731b4f-9164-408f-a14d-229567170101","Type":"ContainerStarted","Data":"dac047e688aa91bb1aefbf1c9926548881b5002c72f7f2c558014604bb9c6bbc"}
Sep 18 19:00:00.990954 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.990931    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cc289c-29829300-k8sw7" event={"ID":"d16993b7-7289-4fe1-acc4-5033b094cc73","Type":"ContainerStarted","Data":"14ca6d4d265773da4f6c82f4d1d6ece81bba3a2ebf55a8974961fb6f923ce957"}
Sep 18 19:00:00.992250 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.992225    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9d9f36-29829300-fhd76" event={"ID":"e38f1aa2-184d-43da-9279-da9d682190be","Type":"ContainerStarted","Data":"8b18506459a535455a3a5e566dc1dbc718ebddd0d25d1efa7161f819cbfc2171"}
Sep 18 19:00:00.994072 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.994022    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-298125-29829300-ctd75" event={"ID":"88696862-bcce-4ac3-b840-a541420b28a6","Type":"ContainerStarted","Data":"c6c4b94f9ec708e2ec5501cde184e0cdbce469b59aaa162c7aea93bd15adb20e"}
Sep 18 19:00:00.994072 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.994063    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-298125-29829300-ctd75" event={"ID":"88696862-bcce-4ac3-b840-a541420b28a6","Type":"ContainerStarted","Data":"c001efb8a66747cbab44746daa26c23fdf2ae14a22fa742d0f81ac586c9118bc"}
Sep 18 19:00:00.996131 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.996025    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-880bee-29829300-d665s" event={"ID":"444696b0-826c-4fe2-b096-52ed6c0ade7d","Type":"ContainerStarted","Data":"07a56c7583d10d51877bec2ab21ba1a5618c16a530324a1a359a2c231fafabc7"}
Sep 18 19:00:00.996131 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.996067    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-880bee-29829300-d665s" event={"ID":"444696b0-826c-4fe2-b096-52ed6c0ade7d","Type":"ContainerStarted","Data":"1e65b788eab89d5131087d587467c3d19acf5a58d025a8d4fc21779cac41961d"}
Sep 18 19:00:00.997601 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.997581    2558 generic.go:356] "Generic (PLEG): container finished" podID="17ce291a-62ba-47d6-a4e7-1b4c3ed05980" containerID="b7a4572e1cdfb3186b86eec8fa783ebea93b198b10fd4ee701c59aa3c2b6e121" exitCode=137
Sep 18 19:00:00.997693 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.997618    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-1be2d2"
Sep 18 19:00:00.997772 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:00.997690    2558 scope.go:117] "RemoveContainer" containerID="b7a4572e1cdfb3186b86eec8fa783ebea93b198b10fd4ee701c59aa3c2b6e121"
Sep 18 19:00:00.997772 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:00.997770    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8950-4aa2b2" container="test-curl-8950-4aa2b2"
Sep 18 19:00:01.011937 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.011900    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-298125-29829300-ctd75" podStartSLOduration=1.011886967 podStartE2EDuration="1.011886967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:00:01.008618827 +0000 UTC m=+1260.445197683" watchObservedRunningTime="2026-09-18 19:00:01.011886967 +0000 UTC m=+1260.448465808"
Sep 18 19:00:01.023705 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.023687    2558 scope.go:117] "RemoveContainer" containerID="b7a4572e1cdfb3186b86eec8fa783ebea93b198b10fd4ee701c59aa3c2b6e121"
Sep 18 19:00:01.024068 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:01.024043    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a4572e1cdfb3186b86eec8fa783ebea93b198b10fd4ee701c59aa3c2b6e121\": container with ID starting with b7a4572e1cdfb3186b86eec8fa783ebea93b198b10fd4ee701c59aa3c2b6e121 not found: ID does not exist" containerID="b7a4572e1cdfb3186b86eec8fa783ebea93b198b10fd4ee701c59aa3c2b6e121"
Sep 18 19:00:01.024130 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.024079    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a4572e1cdfb3186b86eec8fa783ebea93b198b10fd4ee701c59aa3c2b6e121"} err="failed to get container status \"b7a4572e1cdfb3186b86eec8fa783ebea93b198b10fd4ee701c59aa3c2b6e121\": rpc error: code = NotFound desc = could not find container \"b7a4572e1cdfb3186b86eec8fa783ebea93b198b10fd4ee701c59aa3c2b6e121\": container with ID starting with b7a4572e1cdfb3186b86eec8fa783ebea93b198b10fd4ee701c59aa3c2b6e121 not found: ID does not exist"
Sep 18 19:00:01.024620 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.024371    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-880bee-29829300-d665s" podStartSLOduration=1.02435781 podStartE2EDuration="1.02435781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:00:01.023995698 +0000 UTC m=+1260.460574542" watchObservedRunningTime="2026-09-18 19:00:01.02435781 +0000 UTC m=+1260.460936645"
Sep 18 19:00:01.523289 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.523214    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"]
Sep 18 19:00:01.523916 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.523899    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6" containerName="test-curl-8950-4aa2b2"
Sep 18 19:00:01.524023 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.524015    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6" containerName="test-curl-8950-4aa2b2"
Sep 18 19:00:01.524178 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.524170    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6" containerName="test-curl-8950-4aa2b2"
Sep 18 19:00:01.527382 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.527367    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.530265 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.530245    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-25dc6358-openshift-default-dockercfg-fq7g5\""
Sep 18 19:00:01.540925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.540901    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"]
Sep 18 19:00:01.594182 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.594149    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-workload-certs\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.594182 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.594206    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znz5s\" (UniqueName: \"kubernetes.io/projected/b066e952-fca5-4b90-ac8b-227ae35b6be4-kube-api-access-znz5s\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.594182 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.594247    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-token\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.594182 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.594262    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-credential-socket\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.594630 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.594363    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-envoy\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.594630 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.594406    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-data\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.594630 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.594463    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-podinfo\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.594630 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.594544    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-workload-socket\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.594630 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.594560    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b066e952-fca5-4b90-ac8b-227ae35b6be4-istiod-ca-cert\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.696026 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.695993    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-workload-socket\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.696260 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.696034    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b066e952-fca5-4b90-ac8b-227ae35b6be4-istiod-ca-cert\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.696260 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.696085    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-workload-certs\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.696260 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.696124    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znz5s\" (UniqueName: \"kubernetes.io/projected/b066e952-fca5-4b90-ac8b-227ae35b6be4-kube-api-access-znz5s\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.696260 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.696163    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-token\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.696260 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.696218    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-credential-socket\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.696536 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.696275    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-envoy\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.696536 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.696309    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-data\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.696536 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.696354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-podinfo\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.696536 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.696392    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-workload-socket\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.697214 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.697039    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-data\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.697214 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.697112    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-workload-certs\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.697214 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.697210    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b066e952-fca5-4b90-ac8b-227ae35b6be4-istiod-ca-cert\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.697461 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.697249    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-credential-socket\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.700222 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.699172    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-podinfo\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.700222 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.699261    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-envoy\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.704834 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.704803    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-token\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.715770 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.715749    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znz5s\" (UniqueName: \"kubernetes.io/projected/b066e952-fca5-4b90-ac8b-227ae35b6be4-kube-api-access-znz5s\") pod \"e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") " pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:01.742922 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.742895    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 18 19:00:01.743021 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.742939    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4" podUID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 18 19:00:01.840014 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:01.839978    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:02.007211 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.007096    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-h6rcs" event={"ID":"9d731b4f-9164-408f-a14d-229567170101","Type":"ContainerStarted","Data":"2819838b5a6f2e2cf79d5f977a75e8b6cd57625473c3b3b75a8b914e4bbc62b5"}
Sep 18 19:00:02.009788 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.009701    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cc289c-29829300-k8sw7" event={"ID":"d16993b7-7289-4fe1-acc4-5033b094cc73","Type":"ContainerStarted","Data":"4480af971734d16e424db84df63ab751e35524c3d09997449c57d889ec32806f"}
Sep 18 19:00:02.012075 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.012036    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9d9f36-29829300-fhd76" event={"ID":"e38f1aa2-184d-43da-9279-da9d682190be","Type":"ContainerStarted","Data":"d859e1a03750e51e43d41e65d5536557c806d8b94ab7bacd7490723e07d7d779"}
Sep 18 19:00:02.014416 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.013882    2558 generic.go:356] "Generic (PLEG): container finished" podID="88696862-bcce-4ac3-b840-a541420b28a6" containerID="c6c4b94f9ec708e2ec5501cde184e0cdbce469b59aaa162c7aea93bd15adb20e" exitCode=0
Sep 18 19:00:02.014416 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.014010    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-298125-29829300-ctd75" event={"ID":"88696862-bcce-4ac3-b840-a541420b28a6","Type":"ContainerDied","Data":"c6c4b94f9ec708e2ec5501cde184e0cdbce469b59aaa162c7aea93bd15adb20e"}
Sep 18 19:00:02.015763 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.015741    2558 generic.go:356] "Generic (PLEG): container finished" podID="444696b0-826c-4fe2-b096-52ed6c0ade7d" containerID="07a56c7583d10d51877bec2ab21ba1a5618c16a530324a1a359a2c231fafabc7" exitCode=0
Sep 18 19:00:02.015864 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.015810    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-880bee-29829300-d665s" event={"ID":"444696b0-826c-4fe2-b096-52ed6c0ade7d","Type":"ContainerDied","Data":"07a56c7583d10d51877bec2ab21ba1a5618c16a530324a1a359a2c231fafabc7"}
Sep 18 19:00:02.016020 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.016002    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8950-4aa2b2" podUID="ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6" containerName="test-curl-8950-4aa2b2" containerID="cri-o://004ce2a588ec4cace5f4169c3f6ca01d7c1476c463605579818493016c52f47d" gracePeriod=2
Sep 18 19:00:02.025519 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.025476    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-h6rcs" podStartSLOduration=2.025461701 podStartE2EDuration="2.025461701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:00:02.022170484 +0000 UTC m=+1261.458749327" watchObservedRunningTime="2026-09-18 19:00:02.025461701 +0000 UTC m=+1261.462040543"
Sep 18 19:00:02.041218 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:02.041156    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8950-4aa2b2\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6" pod="opendatahub/test-curl-8950-4aa2b2"
Sep 18 19:00:02.124168 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.124092    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cc289c-29829300-k8sw7" podStartSLOduration=2.123526777 podStartE2EDuration="2.123526777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:00:02.11768519 +0000 UTC m=+1261.554264034" watchObservedRunningTime="2026-09-18 19:00:02.123526777 +0000 UTC m=+1261.560105621"
Sep 18 19:00:02.176101 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.176045    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9d9f36-29829300-fhd76" podStartSLOduration=2.176023466 podStartE2EDuration="2.176023466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:00:02.145946396 +0000 UTC m=+1261.582525239" watchObservedRunningTime="2026-09-18 19:00:02.176023466 +0000 UTC m=+1261.612602309"
Sep 18 19:00:02.194894 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.192786    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"]
Sep 18 19:00:02.200028 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.199990    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:00:02.200142 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.200058    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:00:02.200142 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.200086    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:00:02.426281 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.426254    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-c50cb7"
Sep 18 19:00:02.428518 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:02.428495    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8950-4aa2b2\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6" pod="opendatahub/test-curl-8950-4aa2b2"
Sep 18 19:00:02.506291 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.506256    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgvdx\" (UniqueName: \"kubernetes.io/projected/3b223294-abd5-41ff-af65-7dce203efdc2-kube-api-access-pgvdx\") pod \"3b223294-abd5-41ff-af65-7dce203efdc2\" (UID: \"3b223294-abd5-41ff-af65-7dce203efdc2\") "
Sep 18 19:00:02.508312 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.508284    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b223294-abd5-41ff-af65-7dce203efdc2-kube-api-access-pgvdx" (OuterVolumeSpecName: "kube-api-access-pgvdx") pod "3b223294-abd5-41ff-af65-7dce203efdc2" (UID: "3b223294-abd5-41ff-af65-7dce203efdc2"). InnerVolumeSpecName "kube-api-access-pgvdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:00:02.582927 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.582894    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b223294-abd5-41ff-af65-7dce203efdc2" path="/var/lib/kubelet/pods/3b223294-abd5-41ff-af65-7dce203efdc2/volumes"
Sep 18 19:00:02.587681 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:02.587654    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8950-4aa2b2\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6" pod="opendatahub/test-curl-8950-4aa2b2"
Sep 18 19:00:02.607664 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.607636    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgvdx\" (UniqueName: \"kubernetes.io/projected/3b223294-abd5-41ff-af65-7dce203efdc2-kube-api-access-pgvdx\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:02.742863 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.742787    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 18 19:00:02.742863 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.742837    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4" podUID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 18 19:00:02.760079 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:02.760038    2558 scope.go:117] "RemoveContainer" containerID="515f6c10bf18541e6fc521124458e4e7f5d87f2400d053103b26d40f01936164"
Sep 18 19:00:03.024378 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.024281    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" event={"ID":"b066e952-fca5-4b90-ac8b-227ae35b6be4","Type":"ContainerStarted","Data":"aa81fa213a363b60b832397aacc8b79c9898b7e15c0025eebf4bc6f045547c98"}
Sep 18 19:00:03.024378 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.024344    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" event={"ID":"b066e952-fca5-4b90-ac8b-227ae35b6be4","Type":"ContainerStarted","Data":"9ad1feb1f20043b18d72d91a1cfaf4cc454aa96d673aa804a99e617732aebf0d"}
Sep 18 19:00:03.026049 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.026024    2558 generic.go:356] "Generic (PLEG): container finished" podID="e38f1aa2-184d-43da-9279-da9d682190be" containerID="d859e1a03750e51e43d41e65d5536557c806d8b94ab7bacd7490723e07d7d779" exitCode=0
Sep 18 19:00:03.026157 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.026102    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9d9f36-29829300-fhd76" event={"ID":"e38f1aa2-184d-43da-9279-da9d682190be","Type":"ContainerDied","Data":"d859e1a03750e51e43d41e65d5536557c806d8b94ab7bacd7490723e07d7d779"}
Sep 18 19:00:03.027383 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.027356    2558 generic.go:356] "Generic (PLEG): container finished" podID="3b223294-abd5-41ff-af65-7dce203efdc2" containerID="a4eb4bf3a96105072e9f2c46e2fdb3d7d0d62635520836f20391a8d90086a5de" exitCode=137
Sep 18 19:00:03.027464 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.027389    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-c50cb7"
Sep 18 19:00:03.027464 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.027450    2558 scope.go:117] "RemoveContainer" containerID="a4eb4bf3a96105072e9f2c46e2fdb3d7d0d62635520836f20391a8d90086a5de"
Sep 18 19:00:03.029635 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.029611    2558 generic.go:356] "Generic (PLEG): container finished" podID="9d731b4f-9164-408f-a14d-229567170101" containerID="2819838b5a6f2e2cf79d5f977a75e8b6cd57625473c3b3b75a8b914e4bbc62b5" exitCode=0
Sep 18 19:00:03.029731 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.029712    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-h6rcs" event={"ID":"9d731b4f-9164-408f-a14d-229567170101","Type":"ContainerDied","Data":"2819838b5a6f2e2cf79d5f977a75e8b6cd57625473c3b3b75a8b914e4bbc62b5"}
Sep 18 19:00:03.031465 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.031449    2558 generic.go:356] "Generic (PLEG): container finished" podID="d16993b7-7289-4fe1-acc4-5033b094cc73" containerID="4480af971734d16e424db84df63ab751e35524c3d09997449c57d889ec32806f" exitCode=0
Sep 18 19:00:03.031542 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.031520    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cc289c-29829300-k8sw7" event={"ID":"d16993b7-7289-4fe1-acc4-5033b094cc73","Type":"ContainerDied","Data":"4480af971734d16e424db84df63ab751e35524c3d09997449c57d889ec32806f"}
Sep 18 19:00:03.037670 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.037648    2558 scope.go:117] "RemoveContainer" containerID="a4eb4bf3a96105072e9f2c46e2fdb3d7d0d62635520836f20391a8d90086a5de"
Sep 18 19:00:03.037986 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:03.037956    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4eb4bf3a96105072e9f2c46e2fdb3d7d0d62635520836f20391a8d90086a5de\": container with ID starting with a4eb4bf3a96105072e9f2c46e2fdb3d7d0d62635520836f20391a8d90086a5de not found: ID does not exist" containerID="a4eb4bf3a96105072e9f2c46e2fdb3d7d0d62635520836f20391a8d90086a5de"
Sep 18 19:00:03.038087 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.037997    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4eb4bf3a96105072e9f2c46e2fdb3d7d0d62635520836f20391a8d90086a5de"} err="failed to get container status \"a4eb4bf3a96105072e9f2c46e2fdb3d7d0d62635520836f20391a8d90086a5de\": rpc error: code = NotFound desc = could not find container \"a4eb4bf3a96105072e9f2c46e2fdb3d7d0d62635520836f20391a8d90086a5de\": container with ID starting with a4eb4bf3a96105072e9f2c46e2fdb3d7d0d62635520836f20391a8d90086a5de not found: ID does not exist"
Sep 18 19:00:03.063205 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.061866    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" podStartSLOduration=2.061841709 podStartE2EDuration="2.061841709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:00:03.044342971 +0000 UTC m=+1262.480921827" watchObservedRunningTime="2026-09-18 19:00:03.061841709 +0000 UTC m=+1262.498420552"
Sep 18 19:00:03.213556 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.213518    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-298125-29829300-ctd75"
Sep 18 19:00:03.257002 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.256798    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-880bee-29829300-d665s"
Sep 18 19:00:03.745951 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.745920    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 19:00:03.746252 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.746228    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 19:00:03.746997 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.746977    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 19:00:03.840578 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.840545    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:03.846479 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:03.846455    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:04.041123 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.041100    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-880bee-29829300-d665s"
Sep 18 19:00:04.041449 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.041127    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-880bee-29829300-d665s" event={"ID":"444696b0-826c-4fe2-b096-52ed6c0ade7d","Type":"ContainerDied","Data":"1e65b788eab89d5131087d587467c3d19acf5a58d025a8d4fc21779cac41961d"}
Sep 18 19:00:04.041449 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.041174    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e65b788eab89d5131087d587467c3d19acf5a58d025a8d4fc21779cac41961d"
Sep 18 19:00:04.042633 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.042613    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-298125-29829300-ctd75" event={"ID":"88696862-bcce-4ac3-b840-a541420b28a6","Type":"ContainerDied","Data":"c001efb8a66747cbab44746daa26c23fdf2ae14a22fa742d0f81ac586c9118bc"}
Sep 18 19:00:04.042752 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.042630    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-298125-29829300-ctd75"
Sep 18 19:00:04.042752 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.042651    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c001efb8a66747cbab44746daa26c23fdf2ae14a22fa742d0f81ac586c9118bc"
Sep 18 19:00:04.044925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.044901    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:04.047234 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.047172    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:04.244811 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.244785    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-h6rcs"
Sep 18 19:00:04.303841 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.303815    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9d9f36-29829300-fhd76"
Sep 18 19:00:04.306989 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.306968    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-4aa2b2"
Sep 18 19:00:04.310807 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.310790    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cc289c-29829300-k8sw7"
Sep 18 19:00:04.325631 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.325600    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wsgr\" (UniqueName: \"kubernetes.io/projected/ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6-kube-api-access-6wsgr\") pod \"ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6\" (UID: \"ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6\") "
Sep 18 19:00:04.331804 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.331772    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6-kube-api-access-6wsgr" (OuterVolumeSpecName: "kube-api-access-6wsgr") pod "ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6" (UID: "ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6"). InnerVolumeSpecName "kube-api-access-6wsgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:00:04.427860 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.427824    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wsgr\" (UniqueName: \"kubernetes.io/projected/ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6-kube-api-access-6wsgr\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:04.576198 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:04.576113    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6" path="/var/lib/kubelet/pods/ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6/volumes"
Sep 18 19:00:05.053728 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:05.053695    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-h6rcs"
Sep 18 19:00:05.053728 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:05.053715    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-h6rcs" event={"ID":"9d731b4f-9164-408f-a14d-229567170101","Type":"ContainerDied","Data":"dac047e688aa91bb1aefbf1c9926548881b5002c72f7f2c558014604bb9c6bbc"}
Sep 18 19:00:05.054255 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:05.053769    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dac047e688aa91bb1aefbf1c9926548881b5002c72f7f2c558014604bb9c6bbc"
Sep 18 19:00:05.055324 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:05.055308    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cc289c-29829300-k8sw7"
Sep 18 19:00:05.055439 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:05.055350    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cc289c-29829300-k8sw7" event={"ID":"d16993b7-7289-4fe1-acc4-5033b094cc73","Type":"ContainerDied","Data":"14ca6d4d265773da4f6c82f4d1d6ece81bba3a2ebf55a8974961fb6f923ce957"}
Sep 18 19:00:05.055439 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:05.055410    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14ca6d4d265773da4f6c82f4d1d6ece81bba3a2ebf55a8974961fb6f923ce957"
Sep 18 19:00:05.057036 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:05.057007    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9d9f36-29829300-fhd76"
Sep 18 19:00:05.057140 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:05.057058    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9d9f36-29829300-fhd76" event={"ID":"e38f1aa2-184d-43da-9279-da9d682190be","Type":"ContainerDied","Data":"8b18506459a535455a3a5e566dc1dbc718ebddd0d25d1efa7161f819cbfc2171"}
Sep 18 19:00:05.057140 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:05.057106    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b18506459a535455a3a5e566dc1dbc718ebddd0d25d1efa7161f819cbfc2171"
Sep 18 19:00:05.058558 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:05.058536    2558 generic.go:356] "Generic (PLEG): container finished" podID="ede3f37c-8d4a-4a70-8a59-d5e2b7e65bb6" containerID="004ce2a588ec4cace5f4169c3f6ca01d7c1476c463605579818493016c52f47d" exitCode=137
Sep 18 19:00:05.058632 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:05.058566    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8950-4aa2b2"
Sep 18 19:00:05.058632 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:05.058617    2558 scope.go:117] "RemoveContainer" containerID="004ce2a588ec4cace5f4169c3f6ca01d7c1476c463605579818493016c52f47d"
Sep 18 19:00:05.068827 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:05.068810    2558 scope.go:117] "RemoveContainer" containerID="004ce2a588ec4cace5f4169c3f6ca01d7c1476c463605579818493016c52f47d"
Sep 18 19:00:05.069080 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:05.069062    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004ce2a588ec4cace5f4169c3f6ca01d7c1476c463605579818493016c52f47d\": container with ID starting with 004ce2a588ec4cace5f4169c3f6ca01d7c1476c463605579818493016c52f47d not found: ID does not exist" containerID="004ce2a588ec4cace5f4169c3f6ca01d7c1476c463605579818493016c52f47d"
Sep 18 19:00:05.069128 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:05.069087    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004ce2a588ec4cace5f4169c3f6ca01d7c1476c463605579818493016c52f47d"} err="failed to get container status \"004ce2a588ec4cace5f4169c3f6ca01d7c1476c463605579818493016c52f47d\": rpc error: code = NotFound desc = could not find container \"004ce2a588ec4cace5f4169c3f6ca01d7c1476c463605579818493016c52f47d\": container with ID starting with 004ce2a588ec4cace5f4169c3f6ca01d7c1476c463605579818493016c52f47d not found: ID does not exist"
Sep 18 19:00:07.267588 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:07.267554    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b223294_abd5_41ff_af65_7dce203efdc2.slice/crio-a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764\": RecentStats: unable to find data in memory cache]"
Sep 18 19:00:09.885559 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:09.885521    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b223294_abd5_41ff_af65_7dce203efdc2.slice/crio-a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764\": RecentStats: unable to find data in memory cache]"
Sep 18 19:00:09.886009 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:09.885983    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b223294_abd5_41ff_af65_7dce203efdc2.slice/crio-a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764\": RecentStats: unable to find data in memory cache]"
Sep 18 19:00:10.267297 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:10.267173    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b223294_abd5_41ff_af65_7dce203efdc2.slice/crio-a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764\": RecentStats: unable to find data in memory cache]"
Sep 18 19:00:12.882220 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882167    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl"]
Sep 18 19:00:12.882645 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882630    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d731b4f-9164-408f-a14d-229567170101" containerName="cleanup"
Sep 18 19:00:12.882699 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882646    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d731b4f-9164-408f-a14d-229567170101" containerName="cleanup"
Sep 18 19:00:12.882699 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882656    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88696862-bcce-4ac3-b840-a541420b28a6" containerName="cleanup"
Sep 18 19:00:12.882699 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882662    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="88696862-bcce-4ac3-b840-a541420b28a6" containerName="cleanup"
Sep 18 19:00:12.882699 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882673    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d16993b7-7289-4fe1-acc4-5033b094cc73" containerName="cleanup"
Sep 18 19:00:12.882699 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882685    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d16993b7-7289-4fe1-acc4-5033b094cc73" containerName="cleanup"
Sep 18 19:00:12.882699 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882695    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e38f1aa2-184d-43da-9279-da9d682190be" containerName="cleanup"
Sep 18 19:00:12.882699 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882700    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e38f1aa2-184d-43da-9279-da9d682190be" containerName="cleanup"
Sep 18 19:00:12.882931 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882711    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="444696b0-826c-4fe2-b096-52ed6c0ade7d" containerName="cleanup"
Sep 18 19:00:12.882931 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882716    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="444696b0-826c-4fe2-b096-52ed6c0ade7d" containerName="cleanup"
Sep 18 19:00:12.882931 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882793    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="88696862-bcce-4ac3-b840-a541420b28a6" containerName="cleanup"
Sep 18 19:00:12.882931 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882800    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="444696b0-826c-4fe2-b096-52ed6c0ade7d" containerName="cleanup"
Sep 18 19:00:12.882931 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882808    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d16993b7-7289-4fe1-acc4-5033b094cc73" containerName="cleanup"
Sep 18 19:00:12.882931 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882816    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d731b4f-9164-408f-a14d-229567170101" containerName="cleanup"
Sep 18 19:00:12.882931 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.882822    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e38f1aa2-184d-43da-9279-da9d682190be" containerName="cleanup"
Sep 18 19:00:12.887441 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.887422    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl"
Sep 18 19:00:12.890123 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.890104    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:00:12.890815 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.890796    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 19:00:12.900985 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:12.900965    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl"]
Sep 18 19:00:13.016567 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:13.016535    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c78a835-18ea-430e-a2a3-da4191728235-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl\" (UID: \"1c78a835-18ea-430e-a2a3-da4191728235\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl"
Sep 18 19:00:13.117506 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:13.117478    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c78a835-18ea-430e-a2a3-da4191728235-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl\" (UID: \"1c78a835-18ea-430e-a2a3-da4191728235\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl"
Sep 18 19:00:13.118086 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:13.118065    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c78a835-18ea-430e-a2a3-da4191728235-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl\" (UID: \"1c78a835-18ea-430e-a2a3-da4191728235\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl"
Sep 18 19:00:13.199093 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:13.199002    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl"
Sep 18 19:00:13.482917 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:13.482883    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl"]
Sep 18 19:00:13.484564 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:00:13.484535    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c78a835_18ea_430e_a2a3_da4191728235.slice/crio-a61d4a5bf24a8e8ba601ca3b898e1eb2f96832b9d3466bec5951d02bbb3ffa0d WatchSource:0}: Error finding container a61d4a5bf24a8e8ba601ca3b898e1eb2f96832b9d3466bec5951d02bbb3ffa0d: Status 404 returned error can't find the container with id a61d4a5bf24a8e8ba601ca3b898e1eb2f96832b9d3466bec5951d02bbb3ffa0d
Sep 18 19:00:14.131037 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:14.130994    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl" event={"ID":"1c78a835-18ea-430e-a2a3-da4191728235","Type":"ContainerStarted","Data":"5feb46f7638f5ee1ebad2a5df58a697afb8439b9be48e43ac084d0a06dd9846d"}
Sep 18 19:00:14.131498 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:14.131054    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl" event={"ID":"1c78a835-18ea-430e-a2a3-da4191728235","Type":"ContainerStarted","Data":"a61d4a5bf24a8e8ba601ca3b898e1eb2f96832b9d3466bec5951d02bbb3ffa0d"}
Sep 18 19:00:14.161737 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:14.161689    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl" podStartSLOduration=2.161674019 podStartE2EDuration="2.161674019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:00:14.160712597 +0000 UTC m=+1273.597291450" watchObservedRunningTime="2026-09-18 19:00:14.161674019 +0000 UTC m=+1273.598252906"
Sep 18 19:00:15.139361 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:15.139326    2558 generic.go:356] "Generic (PLEG): container finished" podID="1c78a835-18ea-430e-a2a3-da4191728235" containerID="5feb46f7638f5ee1ebad2a5df58a697afb8439b9be48e43ac084d0a06dd9846d" exitCode=0
Sep 18 19:00:15.139764 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:15.139397    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl" event={"ID":"1c78a835-18ea-430e-a2a3-da4191728235","Type":"ContainerDied","Data":"5feb46f7638f5ee1ebad2a5df58a697afb8439b9be48e43ac084d0a06dd9846d"}
Sep 18 19:00:16.288107 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:16.288084    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl"
Sep 18 19:00:16.350092 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:16.350056    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c78a835-18ea-430e-a2a3-da4191728235-maas-api-service-ca\") pod \"1c78a835-18ea-430e-a2a3-da4191728235\" (UID: \"1c78a835-18ea-430e-a2a3-da4191728235\") "
Sep 18 19:00:16.350416 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:16.350394    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c78a835-18ea-430e-a2a3-da4191728235-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c78a835-18ea-430e-a2a3-da4191728235" (UID: "1c78a835-18ea-430e-a2a3-da4191728235"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:00:16.350512 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:16.350490    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c78a835-18ea-430e-a2a3-da4191728235-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:17.156615 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:17.156583    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl"
Sep 18 19:00:17.156822 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:17.156628    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl" event={"ID":"1c78a835-18ea-430e-a2a3-da4191728235","Type":"ContainerDied","Data":"a61d4a5bf24a8e8ba601ca3b898e1eb2f96832b9d3466bec5951d02bbb3ffa0d"}
Sep 18 19:00:17.156822 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:17.156685    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a61d4a5bf24a8e8ba601ca3b898e1eb2f96832b9d3466bec5951d02bbb3ffa0d"
Sep 18 19:00:17.324161 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:17.324129    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b223294_abd5_41ff_af65_7dce203efdc2.slice/crio-a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764\": RecentStats: unable to find data in memory cache]"
Sep 18 19:00:17.813457 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:17.813419    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"]
Sep 18 19:00:17.841043 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:17.841008    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"]
Sep 18 19:00:25.315354 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:25.315322    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b223294_abd5_41ff_af65_7dce203efdc2.slice/crio-a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764\": RecentStats: unable to find data in memory cache]"
Sep 18 19:00:27.335908 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:27.335870    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b223294_abd5_41ff_af65_7dce203efdc2.slice/crio-a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764\": RecentStats: unable to find data in memory cache]"
Sep 18 19:00:29.122268 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:29.122227    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9d9f36-29829300-fhd76"]
Sep 18 19:00:29.131498 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:29.131459    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9d9f36-29829300-fhd76"]
Sep 18 19:00:29.530160 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:29.530081    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl"]
Sep 18 19:00:29.538552 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:29.538526    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9d9f36-tg4zl"]
Sep 18 19:00:30.577857 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:30.577819    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c78a835-18ea-430e-a2a3-da4191728235" path="/var/lib/kubelet/pods/1c78a835-18ea-430e-a2a3-da4191728235/volumes"
Sep 18 19:00:30.578368 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:30.578348    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e38f1aa2-184d-43da-9279-da9d682190be" path="/var/lib/kubelet/pods/e38f1aa2-184d-43da-9279-da9d682190be/volumes"
Sep 18 19:00:32.833068 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:32.833006    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl" podUID="a62c2704-fde4-459c-8299-f32c41f5a4ac" containerName="main" containerID="cri-o://14801e399ad02c7bd2de0d16ae9edea01b2bb5e1914b4c470151691df25deb03" gracePeriod=45
Sep 18 19:00:32.860120 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:32.860059    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl" podUID="2cfa6716-3dda-4774-b7ae-9339f0657634" containerName="main" containerID="cri-o://c07a17fb1a4f409989bcae283c7779b058759be398e387e4ca47f3442a19b92b" gracePeriod=45
Sep 18 19:00:33.136818 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.136788    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 19:00:33.145703 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.145673    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-tmp-dir\") pod \"a62c2704-fde4-459c-8299-f32c41f5a4ac\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") "
Sep 18 19:00:33.145837 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.145749    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62c2704-fde4-459c-8299-f32c41f5a4ac-tls-certs\") pod \"a62c2704-fde4-459c-8299-f32c41f5a4ac\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") "
Sep 18 19:00:33.145837 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.145787    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-dshm\") pod \"a62c2704-fde4-459c-8299-f32c41f5a4ac\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") "
Sep 18 19:00:33.146027 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.145852    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-home\") pod \"a62c2704-fde4-459c-8299-f32c41f5a4ac\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") "
Sep 18 19:00:33.146027 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.145887    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpr6v\" (UniqueName: \"kubernetes.io/projected/a62c2704-fde4-459c-8299-f32c41f5a4ac-kube-api-access-zpr6v\") pod \"a62c2704-fde4-459c-8299-f32c41f5a4ac\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") "
Sep 18 19:00:33.146027 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.145922    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-model-cache\") pod \"a62c2704-fde4-459c-8299-f32c41f5a4ac\" (UID: \"a62c2704-fde4-459c-8299-f32c41f5a4ac\") "
Sep 18 19:00:33.146027 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.145934    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a62c2704-fde4-459c-8299-f32c41f5a4ac" (UID: "a62c2704-fde4-459c-8299-f32c41f5a4ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:00:33.146325 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.146157    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-home" (OuterVolumeSpecName: "home") pod "a62c2704-fde4-459c-8299-f32c41f5a4ac" (UID: "a62c2704-fde4-459c-8299-f32c41f5a4ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:00:33.146390 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.146336    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-tmp-dir\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:33.146448 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.146391    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-home\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:33.146448 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.146406    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "a62c2704-fde4-459c-8299-f32c41f5a4ac" (UID: "a62c2704-fde4-459c-8299-f32c41f5a4ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:00:33.148532 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.148498    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62c2704-fde4-459c-8299-f32c41f5a4ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a62c2704-fde4-459c-8299-f32c41f5a4ac" (UID: "a62c2704-fde4-459c-8299-f32c41f5a4ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:00:33.148724 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.148699    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-dshm" (OuterVolumeSpecName: "dshm") pod "a62c2704-fde4-459c-8299-f32c41f5a4ac" (UID: "a62c2704-fde4-459c-8299-f32c41f5a4ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:00:33.148799 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.148750    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62c2704-fde4-459c-8299-f32c41f5a4ac-kube-api-access-zpr6v" (OuterVolumeSpecName: "kube-api-access-zpr6v") pod "a62c2704-fde4-459c-8299-f32c41f5a4ac" (UID: "a62c2704-fde4-459c-8299-f32c41f5a4ac"). InnerVolumeSpecName "kube-api-access-zpr6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:00:33.164704 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.164680    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 19:00:33.247236 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.247200    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cfa6716-3dda-4774-b7ae-9339f0657634-tls-certs\") pod \"2cfa6716-3dda-4774-b7ae-9339f0657634\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") "
Sep 18 19:00:33.247438 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.247271    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-tmp-dir\") pod \"2cfa6716-3dda-4774-b7ae-9339f0657634\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") "
Sep 18 19:00:33.247438 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.247323    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-home\") pod \"2cfa6716-3dda-4774-b7ae-9339f0657634\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") "
Sep 18 19:00:33.247438 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.247364    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpxxs\" (UniqueName: \"kubernetes.io/projected/2cfa6716-3dda-4774-b7ae-9339f0657634-kube-api-access-dpxxs\") pod \"2cfa6716-3dda-4774-b7ae-9339f0657634\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") "
Sep 18 19:00:33.247438 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.247390    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-model-cache\") pod \"2cfa6716-3dda-4774-b7ae-9339f0657634\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") "
Sep 18 19:00:33.247733 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.247455    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-dshm\") pod \"2cfa6716-3dda-4774-b7ae-9339f0657634\" (UID: \"2cfa6716-3dda-4774-b7ae-9339f0657634\") "
Sep 18 19:00:33.247915 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.247892    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62c2704-fde4-459c-8299-f32c41f5a4ac-tls-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:33.248001 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.247922    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-dshm\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:33.248001 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.247936    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpr6v\" (UniqueName: \"kubernetes.io/projected/a62c2704-fde4-459c-8299-f32c41f5a4ac-kube-api-access-zpr6v\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:33.248001 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.247949    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62c2704-fde4-459c-8299-f32c41f5a4ac-model-cache\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:33.248165 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.248106    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-model-cache" (OuterVolumeSpecName: "model-cache") pod "2cfa6716-3dda-4774-b7ae-9339f0657634" (UID: "2cfa6716-3dda-4774-b7ae-9339f0657634"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:00:33.248245 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.248181    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-home" (OuterVolumeSpecName: "home") pod "2cfa6716-3dda-4774-b7ae-9339f0657634" (UID: "2cfa6716-3dda-4774-b7ae-9339f0657634"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:00:33.248310 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.248264    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2cfa6716-3dda-4774-b7ae-9339f0657634" (UID: "2cfa6716-3dda-4774-b7ae-9339f0657634"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:00:33.249760 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.249728    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cfa6716-3dda-4774-b7ae-9339f0657634-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2cfa6716-3dda-4774-b7ae-9339f0657634" (UID: "2cfa6716-3dda-4774-b7ae-9339f0657634"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:00:33.250363 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.250149    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cfa6716-3dda-4774-b7ae-9339f0657634-kube-api-access-dpxxs" (OuterVolumeSpecName: "kube-api-access-dpxxs") pod "2cfa6716-3dda-4774-b7ae-9339f0657634" (UID: "2cfa6716-3dda-4774-b7ae-9339f0657634"). InnerVolumeSpecName "kube-api-access-dpxxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:00:33.250444 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.250384    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-dshm" (OuterVolumeSpecName: "dshm") pod "2cfa6716-3dda-4774-b7ae-9339f0657634" (UID: "2cfa6716-3dda-4774-b7ae-9339f0657634"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:00:33.273769 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.273617    2558 generic.go:356] "Generic (PLEG): container finished" podID="2cfa6716-3dda-4774-b7ae-9339f0657634" containerID="c07a17fb1a4f409989bcae283c7779b058759be398e387e4ca47f3442a19b92b" exitCode=0
Sep 18 19:00:33.273769 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.273666    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"
Sep 18 19:00:33.273769 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.273703    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl" event={"ID":"2cfa6716-3dda-4774-b7ae-9339f0657634","Type":"ContainerDied","Data":"c07a17fb1a4f409989bcae283c7779b058759be398e387e4ca47f3442a19b92b"}
Sep 18 19:00:33.273769 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.273754    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl" event={"ID":"2cfa6716-3dda-4774-b7ae-9339f0657634","Type":"ContainerDied","Data":"dd96338167e67aeb5833d9cc1b7c2fbdc798f77f3128944fb454a716715a4437"}
Sep 18 19:00:33.274040 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.273778    2558 scope.go:117] "RemoveContainer" containerID="c07a17fb1a4f409989bcae283c7779b058759be398e387e4ca47f3442a19b92b"
Sep 18 19:00:33.275442 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.275419    2558 generic.go:356] "Generic (PLEG): container finished" podID="a62c2704-fde4-459c-8299-f32c41f5a4ac" containerID="14801e399ad02c7bd2de0d16ae9edea01b2bb5e1914b4c470151691df25deb03" exitCode=0
Sep 18 19:00:33.275556 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.275468    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"
Sep 18 19:00:33.275614 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.275553    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl" event={"ID":"a62c2704-fde4-459c-8299-f32c41f5a4ac","Type":"ContainerDied","Data":"14801e399ad02c7bd2de0d16ae9edea01b2bb5e1914b4c470151691df25deb03"}
Sep 18 19:00:33.275614 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.275591    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl" event={"ID":"a62c2704-fde4-459c-8299-f32c41f5a4ac","Type":"ContainerDied","Data":"bea74ca851d2785571dd7889cd67152f1c97daaf2552dccd1cca003ccf08dd6e"}
Sep 18 19:00:33.289428 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.289402    2558 scope.go:117] "RemoveContainer" containerID="c07a17fb1a4f409989bcae283c7779b058759be398e387e4ca47f3442a19b92b"
Sep 18 19:00:33.289762 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:33.289741    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07a17fb1a4f409989bcae283c7779b058759be398e387e4ca47f3442a19b92b\": container with ID starting with c07a17fb1a4f409989bcae283c7779b058759be398e387e4ca47f3442a19b92b not found: ID does not exist" containerID="c07a17fb1a4f409989bcae283c7779b058759be398e387e4ca47f3442a19b92b"
Sep 18 19:00:33.289828 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.289770    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07a17fb1a4f409989bcae283c7779b058759be398e387e4ca47f3442a19b92b"} err="failed to get container status \"c07a17fb1a4f409989bcae283c7779b058759be398e387e4ca47f3442a19b92b\": rpc error: code = NotFound desc = could not find container \"c07a17fb1a4f409989bcae283c7779b058759be398e387e4ca47f3442a19b92b\": container with ID starting with c07a17fb1a4f409989bcae283c7779b058759be398e387e4ca47f3442a19b92b not found: ID does not exist"
Sep 18 19:00:33.289828 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.289789    2558 scope.go:117] "RemoveContainer" containerID="14801e399ad02c7bd2de0d16ae9edea01b2bb5e1914b4c470151691df25deb03"
Sep 18 19:00:33.330085 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.330062    2558 scope.go:117] "RemoveContainer" containerID="14801e399ad02c7bd2de0d16ae9edea01b2bb5e1914b4c470151691df25deb03"
Sep 18 19:00:33.330402 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:33.330382    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14801e399ad02c7bd2de0d16ae9edea01b2bb5e1914b4c470151691df25deb03\": container with ID starting with 14801e399ad02c7bd2de0d16ae9edea01b2bb5e1914b4c470151691df25deb03 not found: ID does not exist" containerID="14801e399ad02c7bd2de0d16ae9edea01b2bb5e1914b4c470151691df25deb03"
Sep 18 19:00:33.330475 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.330410    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14801e399ad02c7bd2de0d16ae9edea01b2bb5e1914b4c470151691df25deb03"} err="failed to get container status \"14801e399ad02c7bd2de0d16ae9edea01b2bb5e1914b4c470151691df25deb03\": rpc error: code = NotFound desc = could not find container \"14801e399ad02c7bd2de0d16ae9edea01b2bb5e1914b4c470151691df25deb03\": container with ID starting with 14801e399ad02c7bd2de0d16ae9edea01b2bb5e1914b4c470151691df25deb03 not found: ID does not exist"
Sep 18 19:00:33.345402 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.345378    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"]
Sep 18 19:00:33.348813 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.348758    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cfa6716-3dda-4774-b7ae-9339f0657634-tls-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:33.348813 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.348778    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-tmp-dir\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:33.348813 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.348787    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-home\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:33.348813 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.348795    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpxxs\" (UniqueName: \"kubernetes.io/projected/2cfa6716-3dda-4774-b7ae-9339f0657634-kube-api-access-dpxxs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:33.348813 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.348804    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-model-cache\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:33.348813 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.348812    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cfa6716-3dda-4774-b7ae-9339f0657634-dshm\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:33.349285 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.349266    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9d9f36/premium-w3-9d9f36-kserve-845474c47b-c9npl"]
Sep 18 19:00:33.361355 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.361334    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"]
Sep 18 19:00:33.365821 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:33.365801    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9d9f36/simulator-w3-9d9f36-kserve-769969484f-7sgtl"]
Sep 18 19:00:34.047051 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:34.047019    2558 patch_prober.go:28] interesting pod/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Sep 18 19:00:34.047444 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:34.047084    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Sep 18 19:00:34.284219 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:34.284109    2558 generic.go:356] "Generic (PLEG): container finished" podID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerID="aa81fa213a363b60b832397aacc8b79c9898b7e15c0025eebf4bc6f045547c98" exitCode=0
Sep 18 19:00:34.284219 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:34.284203    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" event={"ID":"b066e952-fca5-4b90-ac8b-227ae35b6be4","Type":"ContainerDied","Data":"aa81fa213a363b60b832397aacc8b79c9898b7e15c0025eebf4bc6f045547c98"}
Sep 18 19:00:34.284765 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:34.284738    2558 scope.go:117] "RemoveContainer" containerID="aa81fa213a363b60b832397aacc8b79c9898b7e15c0025eebf4bc6f045547c98"
Sep 18 19:00:34.286091 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:34.286053    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:00:34.286269 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:34.286130    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:00:34.286269 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:34.286161    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:00:34.577022 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:34.576936    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cfa6716-3dda-4774-b7ae-9339f0657634" path="/var/lib/kubelet/pods/2cfa6716-3dda-4774-b7ae-9339f0657634/volumes"
Sep 18 19:00:34.577516 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:34.577494    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a62c2704-fde4-459c-8299-f32c41f5a4ac" path="/var/lib/kubelet/pods/a62c2704-fde4-459c-8299-f32c41f5a4ac/volumes"
Sep 18 19:00:35.295134 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:35.295101    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" event={"ID":"b066e952-fca5-4b90-ac8b-227ae35b6be4","Type":"ContainerStarted","Data":"914da9d76288cf358a1dd2616be0a901af5a52358fd3fb262ceac10092de0730"}
Sep 18 19:00:35.840841 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:35.840807    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:35.841061 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:35.841044    2558 patch_prober.go:28] interesting pod/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Sep 18 19:00:35.841110 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:35.841087    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Sep 18 19:00:36.840484 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:36.840445    2558 patch_prober.go:28] interesting pod/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Sep 18 19:00:36.840903 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:36.840504    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Sep 18 19:00:37.197211 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:37.197099    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"]
Sep 18 19:00:37.197578 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:37.197538    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z" podUID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerName="istio-proxy" containerID="cri-o://0791f34afcb43692b05cbe9b12b91fc1c4e4e109aafc1f80f1d0f3dde3da4ada" gracePeriod=30
Sep 18 19:00:37.409241 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:37.409206    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b223294_abd5_41ff_af65_7dce203efdc2.slice/crio-a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764\": RecentStats: unable to find data in memory cache]"
Sep 18 19:00:37.841232 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:37.841170    2558 patch_prober.go:28] interesting pod/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Sep 18 19:00:37.841650 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:37.841257    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Sep 18 19:00:38.321168 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:38.321087    2558 patch_prober.go:28] interesting pod/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 19:00:38.321168 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:38.321151    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z" podUID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:00:38.840441 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:38.840403    2558 patch_prober.go:28] interesting pod/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Sep 18 19:00:38.840622 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:38.840463    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Sep 18 19:00:39.840378 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:39.840341    2558 patch_prober.go:28] interesting pod/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Sep 18 19:00:39.840802 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:39.840411    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Sep 18 19:00:40.276170 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:40.276069    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b223294_abd5_41ff_af65_7dce203efdc2.slice/crio-a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764\": RecentStats: unable to find data in memory cache]"
Sep 18 19:00:40.841324 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:40.841283    2558 patch_prober.go:28] interesting pod/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Sep 18 19:00:40.841706 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:40.841341    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Sep 18 19:00:41.844383 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:41.844343    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:41.844851 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:41.844752    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:41.845325 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:41.845305    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:00:42.348974 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.348935    2558 generic.go:356] "Generic (PLEG): container finished" podID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerID="0791f34afcb43692b05cbe9b12b91fc1c4e4e109aafc1f80f1d0f3dde3da4ada" exitCode=0
Sep 18 19:00:42.349234 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.349010    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z" event={"ID":"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab","Type":"ContainerDied","Data":"0791f34afcb43692b05cbe9b12b91fc1c4e4e109aafc1f80f1d0f3dde3da4ada"}
Sep 18 19:00:42.349234 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.349090    2558 scope.go:117] "RemoveContainer" containerID="b04bde102f863f93aaf8816675e756f2ab29323de7f709896b5e8e3025f1baac"
Sep 18 19:00:42.451536 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.451512    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 19:00:42.541490 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.541399    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istiod-ca-cert\") pod \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") "
Sep 18 19:00:42.541490 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.541442    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-token\") pod \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") "
Sep 18 19:00:42.541490 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.541480    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-workload-socket\") pod \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") "
Sep 18 19:00:42.541773 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.541508    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-workload-certs\") pod \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") "
Sep 18 19:00:42.541773 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.541545    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gcwb\" (UniqueName: \"kubernetes.io/projected/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-kube-api-access-4gcwb\") pod \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") "
Sep 18 19:00:42.541773 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.541572    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-podinfo\") pod \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") "
Sep 18 19:00:42.541773 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.541598    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-envoy\") pod \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") "
Sep 18 19:00:42.541980 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.541805    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-data\") pod \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") "
Sep 18 19:00:42.541980 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.541834    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" (UID: "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:00:42.541980 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.541849    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-credential-socket\") pod \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\" (UID: \"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab\") "
Sep 18 19:00:42.541980 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.541854    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" (UID: "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:00:42.541980 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.541851    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" (UID: "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:00:42.542168 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.542093    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" (UID: "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:00:42.542418 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.542390    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" (UID: "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:00:42.542547 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.542488    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:42.542547 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.542507    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:42.542547 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.542520    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:42.542547 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.542531    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:42.544036 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.544004    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" (UID: "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 19:00:42.544143 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.544056    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" (UID: "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:00:42.544407 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.544233    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-kube-api-access-4gcwb" (OuterVolumeSpecName: "kube-api-access-4gcwb") pod "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" (UID: "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab"). InnerVolumeSpecName "kube-api-access-4gcwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:00:42.544480 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.544252    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" (UID: "5a468882-6c2c-4d9a-87c5-8e4b3e5868ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:00:42.643699 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.643663    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gcwb\" (UniqueName: \"kubernetes.io/projected/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-kube-api-access-4gcwb\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:42.643699 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.643694    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:42.643699 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.643706    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:42.643945 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.643719    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:42.643945 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:42.643732    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:00:43.357132 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.357099    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"
Sep 18 19:00:43.357638 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.357130    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z" event={"ID":"5a468882-6c2c-4d9a-87c5-8e4b3e5868ab","Type":"ContainerDied","Data":"cb9dde1182837d28bea0f17538dd0d0eeb28695cdf93661d1d63631ff4de5af1"}
Sep 18 19:00:43.357638 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.357219    2558 scope.go:117] "RemoveContainer" containerID="0791f34afcb43692b05cbe9b12b91fc1c4e4e109aafc1f80f1d0f3dde3da4ada"
Sep 18 19:00:43.382283 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.382254    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"]
Sep 18 19:00:43.387363 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.387330    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9d9f36-openshift-default-84fc56f996-lfm5z"]
Sep 18 19:00:43.956895 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.956856    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"]
Sep 18 19:00:43.957317 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.957302    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cfa6716-3dda-4774-b7ae-9339f0657634" containerName="main"
Sep 18 19:00:43.957317 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.957317    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cfa6716-3dda-4774-b7ae-9339f0657634" containerName="main"
Sep 18 19:00:43.957437 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.957330    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c78a835-18ea-430e-a2a3-da4191728235" containerName="revoke-keys"
Sep 18 19:00:43.957437 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.957337    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c78a835-18ea-430e-a2a3-da4191728235" containerName="revoke-keys"
Sep 18 19:00:43.957437 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.957352    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerName="istio-proxy"
Sep 18 19:00:43.957437 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.957357    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerName="istio-proxy"
Sep 18 19:00:43.957437 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.957365    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerName="istio-proxy"
Sep 18 19:00:43.957437 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.957371    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerName="istio-proxy"
Sep 18 19:00:43.957437 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.957378    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62c2704-fde4-459c-8299-f32c41f5a4ac" containerName="main"
Sep 18 19:00:43.957437 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.957383    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62c2704-fde4-459c-8299-f32c41f5a4ac" containerName="main"
Sep 18 19:00:43.957851 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.957453    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c78a835-18ea-430e-a2a3-da4191728235" containerName="revoke-keys"
Sep 18 19:00:43.957851 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.957463    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62c2704-fde4-459c-8299-f32c41f5a4ac" containerName="main"
Sep 18 19:00:43.957851 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.957471    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerName="istio-proxy"
Sep 18 19:00:43.957851 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.957477    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" containerName="istio-proxy"
Sep 18 19:00:43.957851 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.957482    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cfa6716-3dda-4774-b7ae-9339f0657634" containerName="main"
Sep 18 19:00:43.961847 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.961828    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:43.966788 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.966768    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6d509\"/\"auth-test-model-a6d509-kserve-self-signed-certs\""
Sep 18 19:00:43.966788 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.966777    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6d509\"/\"kube-root-ca.crt\""
Sep 18 19:00:43.966955 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.966776    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6d509\"/\"openshift-service-ca.crt\""
Sep 18 19:00:43.966955 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.966811    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6d509\"/\"default-dockercfg-kh6mz\""
Sep 18 19:00:43.978252 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:43.978227    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"]
Sep 18 19:00:44.057276 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.057243    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-tmp-dir\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.057276 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.057281    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-tls-certs\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.057536 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.057348    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-dshm\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.057536 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.057393    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxbgf\" (UniqueName: \"kubernetes.io/projected/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-kube-api-access-wxbgf\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.057536 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.057423    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-home\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.057536 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.057454    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-model-cache\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.158295 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.158260    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-tmp-dir\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.158453 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.158302    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-tls-certs\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.158453 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.158355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-dshm\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.158453 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.158398    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxbgf\" (UniqueName: \"kubernetes.io/projected/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-kube-api-access-wxbgf\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.158453 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.158431    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-home\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.158675 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.158479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-model-cache\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.158746 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.158722    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-tmp-dir\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.158829 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.158805    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-model-cache\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.158892 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.158855    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-home\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.160679 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.160658    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-dshm\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.160813 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.160797    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-tls-certs\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.166904 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.166881    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxbgf\" (UniqueName: \"kubernetes.io/projected/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-kube-api-access-wxbgf\") pod \"auth-test-model-a6d509-kserve-7849ffc6f9-27dwd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.271367 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.271280    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:44.569090 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.569058    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"]
Sep 18 19:00:44.570979 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:00:44.570931    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdaff6ee_282e_437f_af3b_5bdc4a1e13cd.slice/crio-ab9e18280eb676b05d34f8d4d0f34a27c94a1e5a883111c9555dd5416edadad3 WatchSource:0}: Error finding container ab9e18280eb676b05d34f8d4d0f34a27c94a1e5a883111c9555dd5416edadad3: Status 404 returned error can't find the container with id ab9e18280eb676b05d34f8d4d0f34a27c94a1e5a883111c9555dd5416edadad3
Sep 18 19:00:44.578733 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:44.578616    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a468882-6c2c-4d9a-87c5-8e4b3e5868ab" path="/var/lib/kubelet/pods/5a468882-6c2c-4d9a-87c5-8e4b3e5868ab/volumes"
Sep 18 19:00:45.376974 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:45.376938    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd" event={"ID":"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd","Type":"ContainerStarted","Data":"1ca5be14f73409e850a27032fbcedbb83e45de1d859287a03aac46c5544a7bad"}
Sep 18 19:00:45.377147 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:45.376992    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd" event={"ID":"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd","Type":"ContainerStarted","Data":"ab9e18280eb676b05d34f8d4d0f34a27c94a1e5a883111c9555dd5416edadad3"}
Sep 18 19:00:45.429542 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:45.429479    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd" podStartSLOduration=2.429463605 podStartE2EDuration="2.429463605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:00:45.426711744 +0000 UTC m=+1304.863290572" watchObservedRunningTime="2026-09-18 19:00:45.429463605 +0000 UTC m=+1304.866042447"
Sep 18 19:00:47.466866 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:47.466831    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b223294_abd5_41ff_af65_7dce203efdc2.slice/crio-a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764\": RecentStats: unable to find data in memory cache]"
Sep 18 19:00:54.272345 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:54.272306    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:54.272719 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:54.272362    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:54.284875 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:54.284850    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:54.451365 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:54.451339    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:00:55.320220 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:55.320174    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b223294_abd5_41ff_af65_7dce203efdc2.slice/crio-a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764\": RecentStats: unable to find data in memory cache]"
Sep 18 19:00:57.477143 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:00:57.477107    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b223294_abd5_41ff_af65_7dce203efdc2.slice/crio-a7e4cd8bacca1f6be8995b0063242878f37b1eb539848e6bbe30567407d84764\": RecentStats: unable to find data in memory cache]"
Sep 18 19:00:59.558862 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.558824    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"]
Sep 18 19:00:59.562688 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.562664    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.565422 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.565395    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-25dc6358\"/\"openshift-service-ca.crt\""
Sep 18 19:00:59.565533 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.565402    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-25dc6358\"/\"kube-root-ca.crt\""
Sep 18 19:00:59.566336 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.566320    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-25dc6358\"/\"default-dockercfg-b598r\""
Sep 18 19:00:59.566414 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.566322    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-25dc6358\"/\"e2e-lifecycle-model-25dc6358-kserve-self-signed-certs\""
Sep 18 19:00:59.577866 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.577845    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"]
Sep 18 19:00:59.728931 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.728901    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-tls-certs\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.729093 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.728937    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-home\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.729093 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.729001    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmqds\" (UniqueName: \"kubernetes.io/projected/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-kube-api-access-xmqds\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.729093 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.729069    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-tmp-dir\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.729231 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.729202    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-dshm\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.729280 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.729236    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-model-cache\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.830414 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.830329    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-tls-certs\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.830414 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.830375    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-home\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.830617 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.830494    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmqds\" (UniqueName: \"kubernetes.io/projected/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-kube-api-access-xmqds\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.830617 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.830575    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-tmp-dir\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.830701 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.830654    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-dshm\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.830701 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.830677    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-model-cache\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.830823 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.830763    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-home\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.830890 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.830875    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-tmp-dir\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.831058 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.831037    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-model-cache\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.832828 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.832796    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-dshm\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.832993 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.832975    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-tls-certs\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.843554 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.843529    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmqds\" (UniqueName: \"kubernetes.io/projected/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-kube-api-access-xmqds\") pod \"e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") " pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:00:59.872476 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:00:59.872444    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:01:00.158794 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:01:00.158758    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08d3a48b_bf5b_4eea_960b_a5fcf584ff52.slice/crio-2cacf8b004b288a3c2ba878e243bdc31e6295e3b64457eeb58c718fc019ec25d WatchSource:0}: Error finding container 2cacf8b004b288a3c2ba878e243bdc31e6295e3b64457eeb58c718fc019ec25d: Status 404 returned error can't find the container with id 2cacf8b004b288a3c2ba878e243bdc31e6295e3b64457eeb58c718fc019ec25d
Sep 18 19:01:00.160049 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:00.160025    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"]
Sep 18 19:01:00.482823 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:00.482718    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s" event={"ID":"08d3a48b-bf5b-4eea-960b-a5fcf584ff52","Type":"ContainerStarted","Data":"9383806406acffd1bacb55cb4112ad6565c811c70cf5ad90e53a29a87d3a328f"}
Sep 18 19:01:00.482823 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:00.482766    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s" event={"ID":"08d3a48b-bf5b-4eea-960b-a5fcf584ff52","Type":"ContainerStarted","Data":"2cacf8b004b288a3c2ba878e243bdc31e6295e3b64457eeb58c718fc019ec25d"}
Sep 18 19:01:00.504199 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:00.504142    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s" podStartSLOduration=1.50412659 podStartE2EDuration="1.50412659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:01:00.500943878 +0000 UTC m=+1319.937522720" watchObservedRunningTime="2026-09-18 19:01:00.50412659 +0000 UTC m=+1319.940705431"
Sep 18 19:01:02.809603 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:02.809570    2558 scope.go:117] "RemoveContainer" containerID="68e372784c1646fa51cb954a0833cc2ead868db5d6b5c408413f9d83aca65eb6"
Sep 18 19:01:09.872895 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:09.872859    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:01:09.873392 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:09.872918    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:01:09.886066 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:09.886036    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:01:10.577042 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:10.577014    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:01:11.766670 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.766637    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"]
Sep 18 19:01:11.771647 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.771623    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.775256 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.775231    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-cc289c\"/\"embedding-w0-cc289c-kserve-self-signed-certs\""
Sep 18 19:01:11.786707 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.786683    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"]
Sep 18 19:01:11.860882 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.860845    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-home\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.860882 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.860882    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687d380c-51cd-4096-bb14-5932e7149e47-tls-certs\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.861112 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.860958    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-tmp-dir\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.861112 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.861006    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-model-cache\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.861112 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.861044    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-dshm\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.861256 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.861156    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7wcs\" (UniqueName: \"kubernetes.io/projected/687d380c-51cd-4096-bb14-5932e7149e47-kube-api-access-v7wcs\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.962086 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.962053    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7wcs\" (UniqueName: \"kubernetes.io/projected/687d380c-51cd-4096-bb14-5932e7149e47-kube-api-access-v7wcs\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.962322 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.962113    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-home\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.962322 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.962300    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687d380c-51cd-4096-bb14-5932e7149e47-tls-certs\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.962443 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.962385    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-tmp-dir\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.962504 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.962444    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-model-cache\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.962504 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.962472    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-home\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.962504 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.962499    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-dshm\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.962761 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.962737    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-tmp-dir\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.962843 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.962810    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-model-cache\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.964813 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.964789    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687d380c-51cd-4096-bb14-5932e7149e47-tls-certs\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.964914 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.964887    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-dshm\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:11.977480 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:11.977458    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7wcs\" (UniqueName: \"kubernetes.io/projected/687d380c-51cd-4096-bb14-5932e7149e47-kube-api-access-v7wcs\") pod \"embedding-w0-cc289c-kserve-f4b7d9848-z9krk\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") " pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:12.081574 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:12.081470    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:12.361591 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:12.361456    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"]
Sep 18 19:01:12.364079 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:01:12.364049    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod687d380c_51cd_4096_bb14_5932e7149e47.slice/crio-8045a1181cd8732654cd86609fba551974f8b587db858fe183bd6bbf6d3d1d65 WatchSource:0}: Error finding container 8045a1181cd8732654cd86609fba551974f8b587db858fe183bd6bbf6d3d1d65: Status 404 returned error can't find the container with id 8045a1181cd8732654cd86609fba551974f8b587db858fe183bd6bbf6d3d1d65
Sep 18 19:01:12.577907 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:12.577863    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk" event={"ID":"687d380c-51cd-4096-bb14-5932e7149e47","Type":"ContainerStarted","Data":"9526cb4d720abc360a69b58516c7b067e582303fd5163f90a2214b135beaea6c"}
Sep 18 19:01:12.578091 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:12.577917    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk" event={"ID":"687d380c-51cd-4096-bb14-5932e7149e47","Type":"ContainerStarted","Data":"8045a1181cd8732654cd86609fba551974f8b587db858fe183bd6bbf6d3d1d65"}
Sep 18 19:01:12.599345 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:12.599295    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk" podStartSLOduration=1.5992799930000001 podStartE2EDuration="1.599279993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:01:12.597040249 +0000 UTC m=+1332.033619115" watchObservedRunningTime="2026-09-18 19:01:12.599279993 +0000 UTC m=+1332.035858834"
Sep 18 19:01:22.082657 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:22.082556    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:22.082657 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:22.082624    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:22.095664 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:22.095638    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:22.652800 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:22.652771    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:01:45.128641 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:45.128606    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv"]
Sep 18 19:01:45.132416 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:45.132398    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv"
Sep 18 19:01:45.134988 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:45.134967    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 19:01:45.135117 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:45.134967    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:01:45.141319 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:45.141066    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv"]
Sep 18 19:01:45.219373 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:45.219336    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3875c19-935f-455a-99ae-2c753efe7eee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv\" (UID: \"a3875c19-935f-455a-99ae-2c753efe7eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv"
Sep 18 19:01:45.320825 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:45.320794    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3875c19-935f-455a-99ae-2c753efe7eee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv\" (UID: \"a3875c19-935f-455a-99ae-2c753efe7eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv"
Sep 18 19:01:45.321432 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:45.321413    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3875c19-935f-455a-99ae-2c753efe7eee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv\" (UID: \"a3875c19-935f-455a-99ae-2c753efe7eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv"
Sep 18 19:01:45.444620 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:45.444538    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv"
Sep 18 19:01:45.726393 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:45.726368    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv"]
Sep 18 19:01:45.727837 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:01:45.727813    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3875c19_935f_455a_99ae_2c753efe7eee.slice/crio-6bdea019deb524e650856cb295db07a2538ec6dff8a2ca10610300d25419a1c0 WatchSource:0}: Error finding container 6bdea019deb524e650856cb295db07a2538ec6dff8a2ca10610300d25419a1c0: Status 404 returned error can't find the container with id 6bdea019deb524e650856cb295db07a2538ec6dff8a2ca10610300d25419a1c0
Sep 18 19:01:45.805650 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:45.805616    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv" event={"ID":"a3875c19-935f-455a-99ae-2c753efe7eee","Type":"ContainerStarted","Data":"6bdea019deb524e650856cb295db07a2538ec6dff8a2ca10610300d25419a1c0"}
Sep 18 19:01:46.815856 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:46.815773    2558 generic.go:356] "Generic (PLEG): container finished" podID="a3875c19-935f-455a-99ae-2c753efe7eee" containerID="3eaa7dba37fde18b7bc36867b89c00f97c6759d40c22e70281c679f1cf345120" exitCode=0
Sep 18 19:01:46.816225 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:46.815858    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv" event={"ID":"a3875c19-935f-455a-99ae-2c753efe7eee","Type":"ContainerDied","Data":"3eaa7dba37fde18b7bc36867b89c00f97c6759d40c22e70281c679f1cf345120"}
Sep 18 19:01:47.974835 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:47.974807    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv"
Sep 18 19:01:48.051744 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:48.051705    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3875c19-935f-455a-99ae-2c753efe7eee-maas-api-service-ca\") pod \"a3875c19-935f-455a-99ae-2c753efe7eee\" (UID: \"a3875c19-935f-455a-99ae-2c753efe7eee\") "
Sep 18 19:01:48.052116 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:48.052084    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3875c19-935f-455a-99ae-2c753efe7eee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3875c19-935f-455a-99ae-2c753efe7eee" (UID: "a3875c19-935f-455a-99ae-2c753efe7eee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:01:48.153713 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:48.153677    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3875c19-935f-455a-99ae-2c753efe7eee-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:01:48.834160 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:48.834082    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv"
Sep 18 19:01:48.834160 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:48.834121    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv" event={"ID":"a3875c19-935f-455a-99ae-2c753efe7eee","Type":"ContainerDied","Data":"6bdea019deb524e650856cb295db07a2538ec6dff8a2ca10610300d25419a1c0"}
Sep 18 19:01:48.834370 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:01:48.834166    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bdea019deb524e650856cb295db07a2538ec6dff8a2ca10610300d25419a1c0"
Sep 18 19:02:00.957465 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:00.957429    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv"]
Sep 18 19:02:00.960479 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:00.960451    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25dc6358-hz9pv"]
Sep 18 19:02:02.576483 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:02.576452    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3875c19-935f-455a-99ae-2c753efe7eee" path="/var/lib/kubelet/pods/a3875c19-935f-455a-99ae-2c753efe7eee/volumes"
Sep 18 19:02:02.855576 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:02.855539    2558 scope.go:117] "RemoveContainer" containerID="f8d90a601d3b32bd09f4bf7ef7f8b30587b00612abed4db251bebb74ebf95da3"
Sep 18 19:02:02.865160 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:02.865140    2558 scope.go:117] "RemoveContainer" containerID="cecb78b391b0feaaf4f5c6a1c4fb162c766d2a80298342a15c8ac8ffccfcbf76"
Sep 18 19:02:08.912042 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:08.911999    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"]
Sep 18 19:02:14.974686 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:14.974650    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt"]
Sep 18 19:02:14.975114 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:14.975100    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3875c19-935f-455a-99ae-2c753efe7eee" containerName="revoke-keys"
Sep 18 19:02:14.975161 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:14.975116    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3875c19-935f-455a-99ae-2c753efe7eee" containerName="revoke-keys"
Sep 18 19:02:14.975234 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:14.975227    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3875c19-935f-455a-99ae-2c753efe7eee" containerName="revoke-keys"
Sep 18 19:02:14.979887 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:14.979869    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt"
Sep 18 19:02:14.982893 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:14.982661    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:02:14.982893 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:14.982667    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 19:02:14.986338 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:14.986314    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt"]
Sep 18 19:02:15.034241 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:15.034204    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ba4093-c7b6-4fa8-af5c-70dc268b14a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt\" (UID: \"e2ba4093-c7b6-4fa8-af5c-70dc268b14a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt"
Sep 18 19:02:15.135427 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:15.135389    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ba4093-c7b6-4fa8-af5c-70dc268b14a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt\" (UID: \"e2ba4093-c7b6-4fa8-af5c-70dc268b14a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt"
Sep 18 19:02:15.136014 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:15.135994    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ba4093-c7b6-4fa8-af5c-70dc268b14a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt\" (UID: \"e2ba4093-c7b6-4fa8-af5c-70dc268b14a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt"
Sep 18 19:02:15.293344 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:15.293263    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt"
Sep 18 19:02:15.584728 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:15.584704    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt"]
Sep 18 19:02:15.587266 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:02:15.587237    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ba4093_c7b6_4fa8_af5c_70dc268b14a3.slice/crio-331347e34f51d4064369654050dba4be200a05659b885b93bfa5f9ed217b7f79 WatchSource:0}: Error finding container 331347e34f51d4064369654050dba4be200a05659b885b93bfa5f9ed217b7f79: Status 404 returned error can't find the container with id 331347e34f51d4064369654050dba4be200a05659b885b93bfa5f9ed217b7f79
Sep 18 19:02:16.020398 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:16.020366    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt" event={"ID":"e2ba4093-c7b6-4fa8-af5c-70dc268b14a3","Type":"ContainerStarted","Data":"5e94745cd09ed17ba6c68ff09040b24c63b612b332dad067f85208a73eecfc50"}
Sep 18 19:02:16.020945 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:16.020424    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt" event={"ID":"e2ba4093-c7b6-4fa8-af5c-70dc268b14a3","Type":"ContainerStarted","Data":"331347e34f51d4064369654050dba4be200a05659b885b93bfa5f9ed217b7f79"}
Sep 18 19:02:16.037618 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:16.037571    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt" podStartSLOduration=2.037555772 podStartE2EDuration="2.037555772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:02:16.035549272 +0000 UTC m=+1395.472128115" watchObservedRunningTime="2026-09-18 19:02:16.037555772 +0000 UTC m=+1395.474134613"
Sep 18 19:02:17.028289 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:17.028253    2558 generic.go:356] "Generic (PLEG): container finished" podID="e2ba4093-c7b6-4fa8-af5c-70dc268b14a3" containerID="5e94745cd09ed17ba6c68ff09040b24c63b612b332dad067f85208a73eecfc50" exitCode=0
Sep 18 19:02:17.028669 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:17.028310    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt" event={"ID":"e2ba4093-c7b6-4fa8-af5c-70dc268b14a3","Type":"ContainerDied","Data":"5e94745cd09ed17ba6c68ff09040b24c63b612b332dad067f85208a73eecfc50"}
Sep 18 19:02:18.187333 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:18.187305    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt"
Sep 18 19:02:18.268837 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:18.268805    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ba4093-c7b6-4fa8-af5c-70dc268b14a3-maas-api-service-ca\") pod \"e2ba4093-c7b6-4fa8-af5c-70dc268b14a3\" (UID: \"e2ba4093-c7b6-4fa8-af5c-70dc268b14a3\") "
Sep 18 19:02:18.269148 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:18.269125    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ba4093-c7b6-4fa8-af5c-70dc268b14a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2ba4093-c7b6-4fa8-af5c-70dc268b14a3" (UID: "e2ba4093-c7b6-4fa8-af5c-70dc268b14a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:02:18.370012 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:18.369974    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ba4093-c7b6-4fa8-af5c-70dc268b14a3-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:19.043892 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:19.043865    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt"
Sep 18 19:02:19.044073 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:19.043896    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt" event={"ID":"e2ba4093-c7b6-4fa8-af5c-70dc268b14a3","Type":"ContainerDied","Data":"331347e34f51d4064369654050dba4be200a05659b885b93bfa5f9ed217b7f79"}
Sep 18 19:02:19.044073 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:19.043954    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="331347e34f51d4064369654050dba4be200a05659b885b93bfa5f9ed217b7f79"
Sep 18 19:02:19.849532 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:19.849495    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"]
Sep 18 19:02:19.865364 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:19.865333    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"]
Sep 18 19:02:19.873133 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:19.873101    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"]
Sep 18 19:02:19.889034 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:19.888999    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"]
Sep 18 19:02:23.931602 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:23.931565    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s" podUID="08d3a48b-bf5b-4eea-960b-a5fcf584ff52" containerName="main" containerID="cri-o://9383806406acffd1bacb55cb4112ad6565c811c70cf5ad90e53a29a87d3a328f" gracePeriod=45
Sep 18 19:02:24.083377 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.083341    2558 generic.go:356] "Generic (PLEG): container finished" podID="08d3a48b-bf5b-4eea-960b-a5fcf584ff52" containerID="9383806406acffd1bacb55cb4112ad6565c811c70cf5ad90e53a29a87d3a328f" exitCode=0
Sep 18 19:02:24.083560 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.083415    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s" event={"ID":"08d3a48b-bf5b-4eea-960b-a5fcf584ff52","Type":"ContainerDied","Data":"9383806406acffd1bacb55cb4112ad6565c811c70cf5ad90e53a29a87d3a328f"}
Sep 18 19:02:24.190758 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.190699    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:02:24.333499 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.333464    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-model-cache\") pod \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") "
Sep 18 19:02:24.333692 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.333509    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-dshm\") pod \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") "
Sep 18 19:02:24.333692 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.333548    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmqds\" (UniqueName: \"kubernetes.io/projected/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-kube-api-access-xmqds\") pod \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") "
Sep 18 19:02:24.333692 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.333647    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-home\") pod \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") "
Sep 18 19:02:24.333854 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.333728    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-tmp-dir\") pod \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") "
Sep 18 19:02:24.333854 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.333746    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-model-cache" (OuterVolumeSpecName: "model-cache") pod "08d3a48b-bf5b-4eea-960b-a5fcf584ff52" (UID: "08d3a48b-bf5b-4eea-960b-a5fcf584ff52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:24.333854 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.333760    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-tls-certs\") pod \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\" (UID: \"08d3a48b-bf5b-4eea-960b-a5fcf584ff52\") "
Sep 18 19:02:24.334028 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.333926    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08d3a48b-bf5b-4eea-960b-a5fcf584ff52" (UID: "08d3a48b-bf5b-4eea-960b-a5fcf584ff52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:24.334028 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.333938    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-home" (OuterVolumeSpecName: "home") pod "08d3a48b-bf5b-4eea-960b-a5fcf584ff52" (UID: "08d3a48b-bf5b-4eea-960b-a5fcf584ff52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:24.334311 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.334276    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-model-cache\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:24.334311 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.334300    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-home\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:24.334311 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.334312    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-tmp-dir\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:24.335837 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.335810    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-dshm" (OuterVolumeSpecName: "dshm") pod "08d3a48b-bf5b-4eea-960b-a5fcf584ff52" (UID: "08d3a48b-bf5b-4eea-960b-a5fcf584ff52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:24.335935 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.335881    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08d3a48b-bf5b-4eea-960b-a5fcf584ff52" (UID: "08d3a48b-bf5b-4eea-960b-a5fcf584ff52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:02:24.335935 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.335915    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-kube-api-access-xmqds" (OuterVolumeSpecName: "kube-api-access-xmqds") pod "08d3a48b-bf5b-4eea-960b-a5fcf584ff52" (UID: "08d3a48b-bf5b-4eea-960b-a5fcf584ff52"). InnerVolumeSpecName "kube-api-access-xmqds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:02:24.435547 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.435518    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-tls-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:24.435547 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.435544    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-dshm\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:24.435547 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:24.435555    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmqds\" (UniqueName: \"kubernetes.io/projected/08d3a48b-bf5b-4eea-960b-a5fcf584ff52-kube-api-access-xmqds\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:25.091992 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:25.091958    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"
Sep 18 19:02:25.092513 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:25.092001    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s" event={"ID":"08d3a48b-bf5b-4eea-960b-a5fcf584ff52","Type":"ContainerDied","Data":"2cacf8b004b288a3c2ba878e243bdc31e6295e3b64457eeb58c718fc019ec25d"}
Sep 18 19:02:25.092513 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:25.092055    2558 scope.go:117] "RemoveContainer" containerID="9383806406acffd1bacb55cb4112ad6565c811c70cf5ad90e53a29a87d3a328f"
Sep 18 19:02:25.109471 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:25.109445    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"]
Sep 18 19:02:25.114636 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:25.114614    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-25dc6358/e2e-lifecycle-model-25dc6358-kserve-6c684fd799-ffv9s"]
Sep 18 19:02:26.575809 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:26.575772    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08d3a48b-bf5b-4eea-960b-a5fcf584ff52" path="/var/lib/kubelet/pods/08d3a48b-bf5b-4eea-960b-a5fcf584ff52/volumes"
Sep 18 19:02:30.607645 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:30.607609    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cc289c-29829300-k8sw7"]
Sep 18 19:02:30.610773 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:30.610746    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cc289c-29829300-k8sw7"]
Sep 18 19:02:30.839856 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:30.839786    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt"]
Sep 18 19:02:30.845639 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:30.845602    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-cc289c-wvnjt"]
Sep 18 19:02:32.580056 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:32.580024    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d16993b7-7289-4fe1-acc4-5033b094cc73" path="/var/lib/kubelet/pods/d16993b7-7289-4fe1-acc4-5033b094cc73/volumes"
Sep 18 19:02:32.580468 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:32.580398    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2ba4093-c7b6-4fa8-af5c-70dc268b14a3" path="/var/lib/kubelet/pods/e2ba4093-c7b6-4fa8-af5c-70dc268b14a3/volumes"
Sep 18 19:02:34.872327 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:34.872268    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv" podUID="9ae3183f-87f3-4891-8aef-7f85a6db3ccf" containerName="main" containerID="cri-o://df0cf48996456c3450eb512115435814c8efe7fd914a120cf9a6674c53df238d" gracePeriod=45
Sep 18 19:02:34.887713 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:34.887627    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk" podUID="687d380c-51cd-4096-bb14-5932e7149e47" containerName="main" containerID="cri-o://9526cb4d720abc360a69b58516c7b067e582303fd5163f90a2214b135beaea6c" gracePeriod=45
Sep 18 19:02:34.894737 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:34.894709    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x" podUID="ce76d297-042d-4fb1-8f79-dbe1ccfb546f" containerName="main" containerID="cri-o://6a0886aa96ca2dd6f177835d84ae28436237455ac20400658f3cfa671323295a" gracePeriod=45
Sep 18 19:02:34.910389 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:34.910359    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl" podUID="c56ef4d9-cf5b-4f41-9304-4ee529cea356" containerName="main" containerID="cri-o://a2f9c2fec418db4662221000b8cf9b872bd05f03a02e9a7f89af10c420ca0fa1" gracePeriod=45
Sep 18 19:02:35.166058 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.166030    2558 generic.go:356] "Generic (PLEG): container finished" podID="9ae3183f-87f3-4891-8aef-7f85a6db3ccf" containerID="df0cf48996456c3450eb512115435814c8efe7fd914a120cf9a6674c53df238d" exitCode=0
Sep 18 19:02:35.166213 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.166114    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv" event={"ID":"9ae3183f-87f3-4891-8aef-7f85a6db3ccf","Type":"ContainerDied","Data":"df0cf48996456c3450eb512115435814c8efe7fd914a120cf9a6674c53df238d"}
Sep 18 19:02:35.167933 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.167912    2558 generic.go:356] "Generic (PLEG): container finished" podID="687d380c-51cd-4096-bb14-5932e7149e47" containerID="9526cb4d720abc360a69b58516c7b067e582303fd5163f90a2214b135beaea6c" exitCode=0
Sep 18 19:02:35.168068 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.167995    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk" event={"ID":"687d380c-51cd-4096-bb14-5932e7149e47","Type":"ContainerDied","Data":"9526cb4d720abc360a69b58516c7b067e582303fd5163f90a2214b135beaea6c"}
Sep 18 19:02:35.170361 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.170288    2558 generic.go:356] "Generic (PLEG): container finished" podID="ce76d297-042d-4fb1-8f79-dbe1ccfb546f" containerID="6a0886aa96ca2dd6f177835d84ae28436237455ac20400658f3cfa671323295a" exitCode=0
Sep 18 19:02:35.170487 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.170405    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x" event={"ID":"ce76d297-042d-4fb1-8f79-dbe1ccfb546f","Type":"ContainerDied","Data":"6a0886aa96ca2dd6f177835d84ae28436237455ac20400658f3cfa671323295a"}
Sep 18 19:02:35.173264 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.173206    2558 generic.go:356] "Generic (PLEG): container finished" podID="c56ef4d9-cf5b-4f41-9304-4ee529cea356" containerID="a2f9c2fec418db4662221000b8cf9b872bd05f03a02e9a7f89af10c420ca0fa1" exitCode=0
Sep 18 19:02:35.173511 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.173494    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl" event={"ID":"c56ef4d9-cf5b-4f41-9304-4ee529cea356","Type":"ContainerDied","Data":"a2f9c2fec418db4662221000b8cf9b872bd05f03a02e9a7f89af10c420ca0fa1"}
Sep 18 19:02:35.202884 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.202853    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:02:35.231816 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.231792    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 19:02:35.239079 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.239055    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 19:02:35.251891 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.249218    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 19:02:35.354096 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354062    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8ffd\" (UniqueName: \"kubernetes.io/projected/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-kube-api-access-h8ffd\") pod \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") "
Sep 18 19:02:35.354280 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354117    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-home\") pod \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") "
Sep 18 19:02:35.354280 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354149    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56ef4d9-cf5b-4f41-9304-4ee529cea356-tls-certs\") pod \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") "
Sep 18 19:02:35.354280 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354175    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrjfd\" (UniqueName: \"kubernetes.io/projected/c56ef4d9-cf5b-4f41-9304-4ee529cea356-kube-api-access-zrjfd\") pod \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") "
Sep 18 19:02:35.354280 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354230    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-home\") pod \"687d380c-51cd-4096-bb14-5932e7149e47\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") "
Sep 18 19:02:35.354280 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354254    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-model-cache\") pod \"687d380c-51cd-4096-bb14-5932e7149e47\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") "
Sep 18 19:02:35.354539 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354282    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-dshm\") pod \"687d380c-51cd-4096-bb14-5932e7149e47\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") "
Sep 18 19:02:35.354539 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354311    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-tmp-dir\") pod \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") "
Sep 18 19:02:35.354539 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354346    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-dshm\") pod \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") "
Sep 18 19:02:35.354539 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354369    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-model-cache\") pod \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") "
Sep 18 19:02:35.354539 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354400    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-model-cache\") pod \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") "
Sep 18 19:02:35.354539 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354439    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-home\") pod \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") "
Sep 18 19:02:35.354539 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354474    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-tls-certs\") pod \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") "
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354571    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687d380c-51cd-4096-bb14-5932e7149e47-tls-certs\") pod \"687d380c-51cd-4096-bb14-5932e7149e47\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") "
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354603    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-tmp-dir\") pod \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") "
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354634    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55fbx\" (UniqueName: \"kubernetes.io/projected/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-kube-api-access-55fbx\") pod \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") "
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354673    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7wcs\" (UniqueName: \"kubernetes.io/projected/687d380c-51cd-4096-bb14-5932e7149e47-kube-api-access-v7wcs\") pod \"687d380c-51cd-4096-bb14-5932e7149e47\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") "
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354704    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-dshm\") pod \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") "
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354743    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-tls-certs\") pod \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") "
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354805    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-home\") pod \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") "
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354835    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-tmp-dir\") pod \"687d380c-51cd-4096-bb14-5932e7149e47\" (UID: \"687d380c-51cd-4096-bb14-5932e7149e47\") "
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354867    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-model-cache\") pod \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\" (UID: \"ce76d297-042d-4fb1-8f79-dbe1ccfb546f\") "
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354431    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-home" (OuterVolumeSpecName: "home") pod "687d380c-51cd-4096-bb14-5932e7149e47" (UID: "687d380c-51cd-4096-bb14-5932e7149e47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354515    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-model-cache" (OuterVolumeSpecName: "model-cache") pod "687d380c-51cd-4096-bb14-5932e7149e47" (UID: "687d380c-51cd-4096-bb14-5932e7149e47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354624    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ae3183f-87f3-4891-8aef-7f85a6db3ccf" (UID: "9ae3183f-87f3-4891-8aef-7f85a6db3ccf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354696    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-home" (OuterVolumeSpecName: "home") pod "c56ef4d9-cf5b-4f41-9304-4ee529cea356" (UID: "c56ef4d9-cf5b-4f41-9304-4ee529cea356"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354720    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ae3183f-87f3-4891-8aef-7f85a6db3ccf" (UID: "9ae3183f-87f3-4891-8aef-7f85a6db3ccf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354864    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-model-cache" (OuterVolumeSpecName: "model-cache") pod "c56ef4d9-cf5b-4f41-9304-4ee529cea356" (UID: "c56ef4d9-cf5b-4f41-9304-4ee529cea356"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.354925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354924    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-dshm\") pod \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\" (UID: \"9ae3183f-87f3-4891-8aef-7f85a6db3ccf\") "
Sep 18 19:02:35.355740 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.354993    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-tmp-dir\") pod \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\" (UID: \"c56ef4d9-cf5b-4f41-9304-4ee529cea356\") "
Sep 18 19:02:35.355740 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.355111    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-home" (OuterVolumeSpecName: "home") pod "9ae3183f-87f3-4891-8aef-7f85a6db3ccf" (UID: "9ae3183f-87f3-4891-8aef-7f85a6db3ccf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.355740 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.355306    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c56ef4d9-cf5b-4f41-9304-4ee529cea356" (UID: "c56ef4d9-cf5b-4f41-9304-4ee529cea356"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.355740 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.355543    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-home\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.355740 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.355562    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-home\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.355740 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.355575    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-model-cache\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.355740 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.355589    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-tmp-dir\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.355740 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.355602    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-model-cache\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.355740 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.355614    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-model-cache\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.355740 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.355627    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-home\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.355740 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.355639    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-tmp-dir\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.356339 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.356243    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-home" (OuterVolumeSpecName: "home") pod "ce76d297-042d-4fb1-8f79-dbe1ccfb546f" (UID: "ce76d297-042d-4fb1-8f79-dbe1ccfb546f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.357302 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.357276    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "687d380c-51cd-4096-bb14-5932e7149e47" (UID: "687d380c-51cd-4096-bb14-5932e7149e47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.357480 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.357375    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce76d297-042d-4fb1-8f79-dbe1ccfb546f" (UID: "ce76d297-042d-4fb1-8f79-dbe1ccfb546f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.357886 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.357476    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-kube-api-access-h8ffd" (OuterVolumeSpecName: "kube-api-access-h8ffd") pod "9ae3183f-87f3-4891-8aef-7f85a6db3ccf" (UID: "9ae3183f-87f3-4891-8aef-7f85a6db3ccf"). InnerVolumeSpecName "kube-api-access-h8ffd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:02:35.358322 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.358107    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56ef4d9-cf5b-4f41-9304-4ee529cea356-kube-api-access-zrjfd" (OuterVolumeSpecName: "kube-api-access-zrjfd") pod "c56ef4d9-cf5b-4f41-9304-4ee529cea356" (UID: "c56ef4d9-cf5b-4f41-9304-4ee529cea356"). InnerVolumeSpecName "kube-api-access-zrjfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:02:35.358322 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.358219    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-dshm" (OuterVolumeSpecName: "dshm") pod "c56ef4d9-cf5b-4f41-9304-4ee529cea356" (UID: "c56ef4d9-cf5b-4f41-9304-4ee529cea356"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.358540 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.358504    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce76d297-042d-4fb1-8f79-dbe1ccfb546f" (UID: "ce76d297-042d-4fb1-8f79-dbe1ccfb546f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.358540 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.358087    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-dshm" (OuterVolumeSpecName: "dshm") pod "9ae3183f-87f3-4891-8aef-7f85a6db3ccf" (UID: "9ae3183f-87f3-4891-8aef-7f85a6db3ccf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.358947 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.358899    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56ef4d9-cf5b-4f41-9304-4ee529cea356-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c56ef4d9-cf5b-4f41-9304-4ee529cea356" (UID: "c56ef4d9-cf5b-4f41-9304-4ee529cea356"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:02:35.359352 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.359311    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce76d297-042d-4fb1-8f79-dbe1ccfb546f" (UID: "ce76d297-042d-4fb1-8f79-dbe1ccfb546f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:02:35.359352 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.359311    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-dshm" (OuterVolumeSpecName: "dshm") pod "ce76d297-042d-4fb1-8f79-dbe1ccfb546f" (UID: "ce76d297-042d-4fb1-8f79-dbe1ccfb546f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.359352 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.359317    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-dshm" (OuterVolumeSpecName: "dshm") pod "687d380c-51cd-4096-bb14-5932e7149e47" (UID: "687d380c-51cd-4096-bb14-5932e7149e47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:35.359644 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.359616    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/687d380c-51cd-4096-bb14-5932e7149e47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "687d380c-51cd-4096-bb14-5932e7149e47" (UID: "687d380c-51cd-4096-bb14-5932e7149e47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:02:35.360154 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.360139    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687d380c-51cd-4096-bb14-5932e7149e47-kube-api-access-v7wcs" (OuterVolumeSpecName: "kube-api-access-v7wcs") pod "687d380c-51cd-4096-bb14-5932e7149e47" (UID: "687d380c-51cd-4096-bb14-5932e7149e47"). InnerVolumeSpecName "kube-api-access-v7wcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:02:35.360246 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.360232    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ae3183f-87f3-4891-8aef-7f85a6db3ccf" (UID: "9ae3183f-87f3-4891-8aef-7f85a6db3ccf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:02:35.360578 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.360557    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-kube-api-access-55fbx" (OuterVolumeSpecName: "kube-api-access-55fbx") pod "ce76d297-042d-4fb1-8f79-dbe1ccfb546f" (UID: "ce76d297-042d-4fb1-8f79-dbe1ccfb546f"). InnerVolumeSpecName "kube-api-access-55fbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:02:35.457233 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457111    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-dshm\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.457233 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457143    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-tls-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.457233 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457157    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-home\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.457233 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457170    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-tmp-dir\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.457233 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457180    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-model-cache\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.457233 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457219    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-dshm\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.457233 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457231    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8ffd\" (UniqueName: \"kubernetes.io/projected/9ae3183f-87f3-4891-8aef-7f85a6db3ccf-kube-api-access-h8ffd\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.457652 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457245    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56ef4d9-cf5b-4f41-9304-4ee529cea356-tls-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.457652 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457258    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrjfd\" (UniqueName: \"kubernetes.io/projected/c56ef4d9-cf5b-4f41-9304-4ee529cea356-kube-api-access-zrjfd\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.457652 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457268    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/687d380c-51cd-4096-bb14-5932e7149e47-dshm\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.457652 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457279    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c56ef4d9-cf5b-4f41-9304-4ee529cea356-dshm\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.457652 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457290    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-tls-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.457652 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457300    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687d380c-51cd-4096-bb14-5932e7149e47-tls-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.457652 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457310    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-tmp-dir\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.457652 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457322    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55fbx\" (UniqueName: \"kubernetes.io/projected/ce76d297-042d-4fb1-8f79-dbe1ccfb546f-kube-api-access-55fbx\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:35.457652 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:35.457335    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7wcs\" (UniqueName: \"kubernetes.io/projected/687d380c-51cd-4096-bb14-5932e7149e47-kube-api-access-v7wcs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:36.181652 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.181561    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"
Sep 18 19:02:36.181652 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.181583    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x" event={"ID":"ce76d297-042d-4fb1-8f79-dbe1ccfb546f","Type":"ContainerDied","Data":"c3007013291cb86f197af3d800a2ee505f3d63698ba0b548a1fe8c2bcd7c8b52"}
Sep 18 19:02:36.181652 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.181632    2558 scope.go:117] "RemoveContainer" containerID="6a0886aa96ca2dd6f177835d84ae28436237455ac20400658f3cfa671323295a"
Sep 18 19:02:36.183897 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.183864    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl" event={"ID":"c56ef4d9-cf5b-4f41-9304-4ee529cea356","Type":"ContainerDied","Data":"68e0a63f71bd4d79aecf5de772ccc4ba4c088e69df0ea3223d9ef424fe6aa210"}
Sep 18 19:02:36.184009 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.183969    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"
Sep 18 19:02:36.197980 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.196590    2558 scope.go:117] "RemoveContainer" containerID="a2f9c2fec418db4662221000b8cf9b872bd05f03a02e9a7f89af10c420ca0fa1"
Sep 18 19:02:36.197980 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.196661    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv" event={"ID":"9ae3183f-87f3-4891-8aef-7f85a6db3ccf","Type":"ContainerDied","Data":"1f046baea2c1927ca45ca1b7c938234b87ac8dfb7a9d2048e4e0e4e24014535f"}
Sep 18 19:02:36.197980 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.196754    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"
Sep 18 19:02:36.202374 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.201732    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk" event={"ID":"687d380c-51cd-4096-bb14-5932e7149e47","Type":"ContainerDied","Data":"8045a1181cd8732654cd86609fba551974f8b587db858fe183bd6bbf6d3d1d65"}
Sep 18 19:02:36.202374 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.201805    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"
Sep 18 19:02:36.218228 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.218179    2558 scope.go:117] "RemoveContainer" containerID="df0cf48996456c3450eb512115435814c8efe7fd914a120cf9a6674c53df238d"
Sep 18 19:02:36.228569 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.228539    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"]
Sep 18 19:02:36.231523 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.231497    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/premium-w0-cc289c-kserve-85d8f7fd8-bjr5x"]
Sep 18 19:02:36.237084 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.236970    2558 scope.go:117] "RemoveContainer" containerID="9526cb4d720abc360a69b58516c7b067e582303fd5163f90a2214b135beaea6c"
Sep 18 19:02:36.240502 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.240482    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"]
Sep 18 19:02:36.244774 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.244751    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/simulator-w0-cc289c-kserve-679db4654f-lpxvl"]
Sep 18 19:02:36.257771 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.257745    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"]
Sep 18 19:02:36.261230 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.261209    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/distinct-2-w0-cc289c-kserve-7cbcb7ccc6-lwvdv"]
Sep 18 19:02:36.269608 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.269583    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"]
Sep 18 19:02:36.273115 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.273095    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-cc289c/embedding-w0-cc289c-kserve-f4b7d9848-z9krk"]
Sep 18 19:02:36.577140 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.577047    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="687d380c-51cd-4096-bb14-5932e7149e47" path="/var/lib/kubelet/pods/687d380c-51cd-4096-bb14-5932e7149e47/volumes"
Sep 18 19:02:36.577605 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.577584    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ae3183f-87f3-4891-8aef-7f85a6db3ccf" path="/var/lib/kubelet/pods/9ae3183f-87f3-4891-8aef-7f85a6db3ccf/volumes"
Sep 18 19:02:36.578038 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.578019    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c56ef4d9-cf5b-4f41-9304-4ee529cea356" path="/var/lib/kubelet/pods/c56ef4d9-cf5b-4f41-9304-4ee529cea356/volumes"
Sep 18 19:02:36.578772 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:36.578752    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce76d297-042d-4fb1-8f79-dbe1ccfb546f" path="/var/lib/kubelet/pods/ce76d297-042d-4fb1-8f79-dbe1ccfb546f/volumes"
Sep 18 19:02:38.147901 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:38.147866    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"]
Sep 18 19:02:38.148399 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:38.148109    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj" podUID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerName="istio-proxy" containerID="cri-o://8438cbb341fcafeca97bb8175c5957e0f8824fcff1f1f12ed4a8c21478e7fd13" gracePeriod=30
Sep 18 19:02:43.406221 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.406180    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 19:02:43.536746 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.536658    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-credential-socket\") pod \"42bf6cd3-467b-4c9f-9960-c1214c476a44\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") "
Sep 18 19:02:43.536746 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.536723    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-data\") pod \"42bf6cd3-467b-4c9f-9960-c1214c476a44\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") "
Sep 18 19:02:43.536991 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.536761    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-token\") pod \"42bf6cd3-467b-4c9f-9960-c1214c476a44\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") "
Sep 18 19:02:43.536991 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.536787    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-workload-socket\") pod \"42bf6cd3-467b-4c9f-9960-c1214c476a44\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") "
Sep 18 19:02:43.536991 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.536843    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-podinfo\") pod \"42bf6cd3-467b-4c9f-9960-c1214c476a44\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") "
Sep 18 19:02:43.536991 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.536870    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-envoy\") pod \"42bf6cd3-467b-4c9f-9960-c1214c476a44\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") "
Sep 18 19:02:43.536991 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.536899    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42bf6cd3-467b-4c9f-9960-c1214c476a44-istiod-ca-cert\") pod \"42bf6cd3-467b-4c9f-9960-c1214c476a44\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") "
Sep 18 19:02:43.536991 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.536957    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9kqx\" (UniqueName: \"kubernetes.io/projected/42bf6cd3-467b-4c9f-9960-c1214c476a44-kube-api-access-l9kqx\") pod \"42bf6cd3-467b-4c9f-9960-c1214c476a44\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") "
Sep 18 19:02:43.536991 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.536957    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "42bf6cd3-467b-4c9f-9960-c1214c476a44" (UID: "42bf6cd3-467b-4c9f-9960-c1214c476a44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:43.537372 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.537016    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-workload-certs\") pod \"42bf6cd3-467b-4c9f-9960-c1214c476a44\" (UID: \"42bf6cd3-467b-4c9f-9960-c1214c476a44\") "
Sep 18 19:02:43.537372 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.537057    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-data" (OuterVolumeSpecName: "istio-data") pod "42bf6cd3-467b-4c9f-9960-c1214c476a44" (UID: "42bf6cd3-467b-4c9f-9960-c1214c476a44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:43.537372 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.537271    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "42bf6cd3-467b-4c9f-9960-c1214c476a44" (UID: "42bf6cd3-467b-4c9f-9960-c1214c476a44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:43.537517 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.537369    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "42bf6cd3-467b-4c9f-9960-c1214c476a44" (UID: "42bf6cd3-467b-4c9f-9960-c1214c476a44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:43.537517 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.537407    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:43.537517 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.537427    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:43.537517 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.537442    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:43.537517 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.537452    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bf6cd3-467b-4c9f-9960-c1214c476a44-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "42bf6cd3-467b-4c9f-9960-c1214c476a44" (UID: "42bf6cd3-467b-4c9f-9960-c1214c476a44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:02:43.539245 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.539218    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bf6cd3-467b-4c9f-9960-c1214c476a44-kube-api-access-l9kqx" (OuterVolumeSpecName: "kube-api-access-l9kqx") pod "42bf6cd3-467b-4c9f-9960-c1214c476a44" (UID: "42bf6cd3-467b-4c9f-9960-c1214c476a44"). InnerVolumeSpecName "kube-api-access-l9kqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:02:43.539245 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.539219    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "42bf6cd3-467b-4c9f-9960-c1214c476a44" (UID: "42bf6cd3-467b-4c9f-9960-c1214c476a44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:43.539522 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.539500    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "42bf6cd3-467b-4c9f-9960-c1214c476a44" (UID: "42bf6cd3-467b-4c9f-9960-c1214c476a44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 19:02:43.539522 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.539512    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-token" (OuterVolumeSpecName: "istio-token") pod "42bf6cd3-467b-4c9f-9960-c1214c476a44" (UID: "42bf6cd3-467b-4c9f-9960-c1214c476a44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:02:43.638786 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.638750    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:43.638786 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.638780    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:43.638786 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.638789    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:43.639015 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.638797    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42bf6cd3-467b-4c9f-9960-c1214c476a44-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:43.639015 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.638805    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9kqx\" (UniqueName: \"kubernetes.io/projected/42bf6cd3-467b-4c9f-9960-c1214c476a44-kube-api-access-l9kqx\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:43.639015 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:43.638814    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42bf6cd3-467b-4c9f-9960-c1214c476a44-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:44.260121 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:44.260086    2558 generic.go:356] "Generic (PLEG): container finished" podID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerID="8438cbb341fcafeca97bb8175c5957e0f8824fcff1f1f12ed4a8c21478e7fd13" exitCode=0
Sep 18 19:02:44.260290 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:44.260155    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"
Sep 18 19:02:44.260290 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:44.260169    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj" event={"ID":"42bf6cd3-467b-4c9f-9960-c1214c476a44","Type":"ContainerDied","Data":"8438cbb341fcafeca97bb8175c5957e0f8824fcff1f1f12ed4a8c21478e7fd13"}
Sep 18 19:02:44.260290 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:44.260246    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj" event={"ID":"42bf6cd3-467b-4c9f-9960-c1214c476a44","Type":"ContainerDied","Data":"74ff2804920fa7df62016ec89ad10c7b0f0c4d3c2b2e7cfb43bb65e9195a778a"}
Sep 18 19:02:44.260290 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:44.260270    2558 scope.go:117] "RemoveContainer" containerID="8438cbb341fcafeca97bb8175c5957e0f8824fcff1f1f12ed4a8c21478e7fd13"
Sep 18 19:02:44.271003 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:44.270821    2558 scope.go:117] "RemoveContainer" containerID="2844f9c5b2702140a5cd8766edf6a93dbe0836d9351485f62b573bd779489c5d"
Sep 18 19:02:44.279865 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:44.279849    2558 scope.go:117] "RemoveContainer" containerID="8438cbb341fcafeca97bb8175c5957e0f8824fcff1f1f12ed4a8c21478e7fd13"
Sep 18 19:02:44.280140 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:02:44.280122    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8438cbb341fcafeca97bb8175c5957e0f8824fcff1f1f12ed4a8c21478e7fd13\": container with ID starting with 8438cbb341fcafeca97bb8175c5957e0f8824fcff1f1f12ed4a8c21478e7fd13 not found: ID does not exist" containerID="8438cbb341fcafeca97bb8175c5957e0f8824fcff1f1f12ed4a8c21478e7fd13"
Sep 18 19:02:44.280236 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:44.280145    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8438cbb341fcafeca97bb8175c5957e0f8824fcff1f1f12ed4a8c21478e7fd13"} err="failed to get container status \"8438cbb341fcafeca97bb8175c5957e0f8824fcff1f1f12ed4a8c21478e7fd13\": rpc error: code = NotFound desc = could not find container \"8438cbb341fcafeca97bb8175c5957e0f8824fcff1f1f12ed4a8c21478e7fd13\": container with ID starting with 8438cbb341fcafeca97bb8175c5957e0f8824fcff1f1f12ed4a8c21478e7fd13 not found: ID does not exist"
Sep 18 19:02:44.280236 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:44.280162    2558 scope.go:117] "RemoveContainer" containerID="2844f9c5b2702140a5cd8766edf6a93dbe0836d9351485f62b573bd779489c5d"
Sep 18 19:02:44.280451 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:02:44.280434    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2844f9c5b2702140a5cd8766edf6a93dbe0836d9351485f62b573bd779489c5d\": container with ID starting with 2844f9c5b2702140a5cd8766edf6a93dbe0836d9351485f62b573bd779489c5d not found: ID does not exist" containerID="2844f9c5b2702140a5cd8766edf6a93dbe0836d9351485f62b573bd779489c5d"
Sep 18 19:02:44.280493 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:44.280459    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2844f9c5b2702140a5cd8766edf6a93dbe0836d9351485f62b573bd779489c5d"} err="failed to get container status \"2844f9c5b2702140a5cd8766edf6a93dbe0836d9351485f62b573bd779489c5d\": rpc error: code = NotFound desc = could not find container \"2844f9c5b2702140a5cd8766edf6a93dbe0836d9351485f62b573bd779489c5d\": container with ID starting with 2844f9c5b2702140a5cd8766edf6a93dbe0836d9351485f62b573bd779489c5d not found: ID does not exist"
Sep 18 19:02:44.286006 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:44.285986    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"]
Sep 18 19:02:44.292398 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:44.292376    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-cc289c-openshift-default-5d8fc56c48-zj6gj"]
Sep 18 19:02:44.577426 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:44.577344    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42bf6cd3-467b-4c9f-9960-c1214c476a44" path="/var/lib/kubelet/pods/42bf6cd3-467b-4c9f-9960-c1214c476a44/volumes"
Sep 18 19:02:51.836283 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:51.836201    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"]
Sep 18 19:02:51.836860 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:51.836472    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerName="istio-proxy" containerID="cri-o://914da9d76288cf358a1dd2616be0a901af5a52358fd3fb262ceac10092de0730" gracePeriod=30
Sep 18 19:02:56.846610 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:56.846577    2558 patch_prober.go:28] interesting pod/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Sep 18 19:02:56.846982 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:56.846644    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Sep 18 19:02:57.092445 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.092420    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:02:57.167101 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167014    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-envoy\") pod \"b066e952-fca5-4b90-ac8b-227ae35b6be4\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") "
Sep 18 19:02:57.167101 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167051    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-podinfo\") pod \"b066e952-fca5-4b90-ac8b-227ae35b6be4\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") "
Sep 18 19:02:57.167101 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167074    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-token\") pod \"b066e952-fca5-4b90-ac8b-227ae35b6be4\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") "
Sep 18 19:02:57.167101 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167099    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znz5s\" (UniqueName: \"kubernetes.io/projected/b066e952-fca5-4b90-ac8b-227ae35b6be4-kube-api-access-znz5s\") pod \"b066e952-fca5-4b90-ac8b-227ae35b6be4\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") "
Sep 18 19:02:57.167491 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167148    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b066e952-fca5-4b90-ac8b-227ae35b6be4-istiod-ca-cert\") pod \"b066e952-fca5-4b90-ac8b-227ae35b6be4\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") "
Sep 18 19:02:57.167491 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167177    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-workload-certs\") pod \"b066e952-fca5-4b90-ac8b-227ae35b6be4\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") "
Sep 18 19:02:57.167491 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167273    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-data\") pod \"b066e952-fca5-4b90-ac8b-227ae35b6be4\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") "
Sep 18 19:02:57.167491 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167301    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-credential-socket\") pod \"b066e952-fca5-4b90-ac8b-227ae35b6be4\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") "
Sep 18 19:02:57.167491 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167364    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-workload-socket\") pod \"b066e952-fca5-4b90-ac8b-227ae35b6be4\" (UID: \"b066e952-fca5-4b90-ac8b-227ae35b6be4\") "
Sep 18 19:02:57.167810 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167520    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b066e952-fca5-4b90-ac8b-227ae35b6be4" (UID: "b066e952-fca5-4b90-ac8b-227ae35b6be4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:57.167810 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167679    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b066e952-fca5-4b90-ac8b-227ae35b6be4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b066e952-fca5-4b90-ac8b-227ae35b6be4" (UID: "b066e952-fca5-4b90-ac8b-227ae35b6be4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:02:57.167810 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167691    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-data" (OuterVolumeSpecName: "istio-data") pod "b066e952-fca5-4b90-ac8b-227ae35b6be4" (UID: "b066e952-fca5-4b90-ac8b-227ae35b6be4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:57.167810 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167719    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b066e952-fca5-4b90-ac8b-227ae35b6be4" (UID: "b066e952-fca5-4b90-ac8b-227ae35b6be4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:57.167810 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167790    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b066e952-fca5-4b90-ac8b-227ae35b6be4-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:57.167810 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167810    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:57.168126 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167822    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:57.168126 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.167854    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b066e952-fca5-4b90-ac8b-227ae35b6be4" (UID: "b066e952-fca5-4b90-ac8b-227ae35b6be4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:57.169484 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.169458    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b066e952-fca5-4b90-ac8b-227ae35b6be4" (UID: "b066e952-fca5-4b90-ac8b-227ae35b6be4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:02:57.169613 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.169506    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-token" (OuterVolumeSpecName: "istio-token") pod "b066e952-fca5-4b90-ac8b-227ae35b6be4" (UID: "b066e952-fca5-4b90-ac8b-227ae35b6be4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:02:57.169727 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.169712    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b066e952-fca5-4b90-ac8b-227ae35b6be4" (UID: "b066e952-fca5-4b90-ac8b-227ae35b6be4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 19:02:57.169788 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.169744    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b066e952-fca5-4b90-ac8b-227ae35b6be4-kube-api-access-znz5s" (OuterVolumeSpecName: "kube-api-access-znz5s") pod "b066e952-fca5-4b90-ac8b-227ae35b6be4" (UID: "b066e952-fca5-4b90-ac8b-227ae35b6be4"). InnerVolumeSpecName "kube-api-access-znz5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:02:57.269125 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.269092    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:57.269125 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.269118    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:57.269125 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.269127    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b066e952-fca5-4b90-ac8b-227ae35b6be4-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:57.269371 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.269135    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znz5s\" (UniqueName: \"kubernetes.io/projected/b066e952-fca5-4b90-ac8b-227ae35b6be4-kube-api-access-znz5s\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:57.269371 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.269149    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:57.269371 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.269157    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b066e952-fca5-4b90-ac8b-227ae35b6be4-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:02:57.346539 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.346501    2558 generic.go:356] "Generic (PLEG): container finished" podID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerID="914da9d76288cf358a1dd2616be0a901af5a52358fd3fb262ceac10092de0730" exitCode=0
Sep 18 19:02:57.346736 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.346556    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"
Sep 18 19:02:57.346736 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.346593    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" event={"ID":"b066e952-fca5-4b90-ac8b-227ae35b6be4","Type":"ContainerDied","Data":"914da9d76288cf358a1dd2616be0a901af5a52358fd3fb262ceac10092de0730"}
Sep 18 19:02:57.346736 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.346651    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn" event={"ID":"b066e952-fca5-4b90-ac8b-227ae35b6be4","Type":"ContainerDied","Data":"9ad1feb1f20043b18d72d91a1cfaf4cc454aa96d673aa804a99e617732aebf0d"}
Sep 18 19:02:57.346736 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.346668    2558 scope.go:117] "RemoveContainer" containerID="914da9d76288cf358a1dd2616be0a901af5a52358fd3fb262ceac10092de0730"
Sep 18 19:02:57.357252 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.357225    2558 scope.go:117] "RemoveContainer" containerID="aa81fa213a363b60b832397aacc8b79c9898b7e15c0025eebf4bc6f045547c98"
Sep 18 19:02:57.366871 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.366850    2558 scope.go:117] "RemoveContainer" containerID="914da9d76288cf358a1dd2616be0a901af5a52358fd3fb262ceac10092de0730"
Sep 18 19:02:57.367205 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:02:57.367164    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914da9d76288cf358a1dd2616be0a901af5a52358fd3fb262ceac10092de0730\": container with ID starting with 914da9d76288cf358a1dd2616be0a901af5a52358fd3fb262ceac10092de0730 not found: ID does not exist" containerID="914da9d76288cf358a1dd2616be0a901af5a52358fd3fb262ceac10092de0730"
Sep 18 19:02:57.367295 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.367222    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914da9d76288cf358a1dd2616be0a901af5a52358fd3fb262ceac10092de0730"} err="failed to get container status \"914da9d76288cf358a1dd2616be0a901af5a52358fd3fb262ceac10092de0730\": rpc error: code = NotFound desc = could not find container \"914da9d76288cf358a1dd2616be0a901af5a52358fd3fb262ceac10092de0730\": container with ID starting with 914da9d76288cf358a1dd2616be0a901af5a52358fd3fb262ceac10092de0730 not found: ID does not exist"
Sep 18 19:02:57.367295 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.367243    2558 scope.go:117] "RemoveContainer" containerID="aa81fa213a363b60b832397aacc8b79c9898b7e15c0025eebf4bc6f045547c98"
Sep 18 19:02:57.367542 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:02:57.367525    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa81fa213a363b60b832397aacc8b79c9898b7e15c0025eebf4bc6f045547c98\": container with ID starting with aa81fa213a363b60b832397aacc8b79c9898b7e15c0025eebf4bc6f045547c98 not found: ID does not exist" containerID="aa81fa213a363b60b832397aacc8b79c9898b7e15c0025eebf4bc6f045547c98"
Sep 18 19:02:57.367592 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.367546    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa81fa213a363b60b832397aacc8b79c9898b7e15c0025eebf4bc6f045547c98"} err="failed to get container status \"aa81fa213a363b60b832397aacc8b79c9898b7e15c0025eebf4bc6f045547c98\": rpc error: code = NotFound desc = could not find container \"aa81fa213a363b60b832397aacc8b79c9898b7e15c0025eebf4bc6f045547c98\": container with ID starting with aa81fa213a363b60b832397aacc8b79c9898b7e15c0025eebf4bc6f045547c98 not found: ID does not exist"
Sep 18 19:02:57.368962 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.368935    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"]
Sep 18 19:02:57.374410 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:57.374387    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-25dc6358-openshift-default-6f9c6979dd-pnznn"]
Sep 18 19:02:58.575939 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:02:58.575906    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" path="/var/lib/kubelet/pods/b066e952-fca5-4b90-ac8b-227ae35b6be4/volumes"
Sep 18 19:03:45.905515 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905474    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8953-607e3f"]
Sep 18 19:03:45.905961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905865    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c56ef4d9-cf5b-4f41-9304-4ee529cea356" containerName="main"
Sep 18 19:03:45.905961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905877    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c56ef4d9-cf5b-4f41-9304-4ee529cea356" containerName="main"
Sep 18 19:03:45.905961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905892    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce76d297-042d-4fb1-8f79-dbe1ccfb546f" containerName="main"
Sep 18 19:03:45.905961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905897    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce76d297-042d-4fb1-8f79-dbe1ccfb546f" containerName="main"
Sep 18 19:03:45.905961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905904    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerName="istio-proxy"
Sep 18 19:03:45.905961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905909    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerName="istio-proxy"
Sep 18 19:03:45.905961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905919    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ba4093-c7b6-4fa8-af5c-70dc268b14a3" containerName="revoke-keys"
Sep 18 19:03:45.905961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905924    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ba4093-c7b6-4fa8-af5c-70dc268b14a3" containerName="revoke-keys"
Sep 18 19:03:45.905961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905930    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae3183f-87f3-4891-8aef-7f85a6db3ccf" containerName="main"
Sep 18 19:03:45.905961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905935    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae3183f-87f3-4891-8aef-7f85a6db3ccf" containerName="main"
Sep 18 19:03:45.905961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905947    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerName="istio-proxy"
Sep 18 19:03:45.905961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905952    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerName="istio-proxy"
Sep 18 19:03:45.905961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905962    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d3a48b-bf5b-4eea-960b-a5fcf584ff52" containerName="main"
Sep 18 19:03:45.905961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905968    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d3a48b-bf5b-4eea-960b-a5fcf584ff52" containerName="main"
Sep 18 19:03:45.906435 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905974    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687d380c-51cd-4096-bb14-5932e7149e47" containerName="main"
Sep 18 19:03:45.906435 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905979    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="687d380c-51cd-4096-bb14-5932e7149e47" containerName="main"
Sep 18 19:03:45.906435 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905985    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerName="istio-proxy"
Sep 18 19:03:45.906435 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.905990    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerName="istio-proxy"
Sep 18 19:03:45.906435 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.906049    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c56ef4d9-cf5b-4f41-9304-4ee529cea356" containerName="main"
Sep 18 19:03:45.906435 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.906056    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ba4093-c7b6-4fa8-af5c-70dc268b14a3" containerName="revoke-keys"
Sep 18 19:03:45.906435 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.906063    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerName="istio-proxy"
Sep 18 19:03:45.906435 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.906068    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ae3183f-87f3-4891-8aef-7f85a6db3ccf" containerName="main"
Sep 18 19:03:45.906435 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.906074    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerName="istio-proxy"
Sep 18 19:03:45.906435 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.906079    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="687d380c-51cd-4096-bb14-5932e7149e47" containerName="main"
Sep 18 19:03:45.906435 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.906087    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce76d297-042d-4fb1-8f79-dbe1ccfb546f" containerName="main"
Sep 18 19:03:45.906435 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.906092    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d3a48b-bf5b-4eea-960b-a5fcf584ff52" containerName="main"
Sep 18 19:03:45.906435 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.906097    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerName="istio-proxy"
Sep 18 19:03:45.909481 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.909463    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8953-607e3f"
Sep 18 19:03:45.912089 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.912070    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7m4wl\""
Sep 18 19:03:45.915983 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:45.915956    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8953-607e3f"]
Sep 18 19:03:46.054130 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.054092    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jmb\" (UniqueName: \"kubernetes.io/projected/98fdb1ec-7158-4908-998a-c31920f58780-kube-api-access-b7jmb\") pod \"test-curl-8953-607e3f\" (UID: \"98fdb1ec-7158-4908-998a-c31920f58780\") " pod="opendatahub/test-curl-8953-607e3f"
Sep 18 19:03:46.155905 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.155803    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jmb\" (UniqueName: \"kubernetes.io/projected/98fdb1ec-7158-4908-998a-c31920f58780-kube-api-access-b7jmb\") pod \"test-curl-8953-607e3f\" (UID: \"98fdb1ec-7158-4908-998a-c31920f58780\") " pod="opendatahub/test-curl-8953-607e3f"
Sep 18 19:03:46.164176 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.164148    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jmb\" (UniqueName: \"kubernetes.io/projected/98fdb1ec-7158-4908-998a-c31920f58780-kube-api-access-b7jmb\") pod \"test-curl-8953-607e3f\" (UID: \"98fdb1ec-7158-4908-998a-c31920f58780\") " pod="opendatahub/test-curl-8953-607e3f"
Sep 18 19:03:46.220410 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.220373    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8953-607e3f"
Sep 18 19:03:46.234261 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.234232    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"]
Sep 18 19:03:46.234693 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.234675    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerName="istio-proxy"
Sep 18 19:03:46.234693 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.234692    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b066e952-fca5-4b90-ac8b-227ae35b6be4" containerName="istio-proxy"
Sep 18 19:03:46.234806 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.234774    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="42bf6cd3-467b-4c9f-9960-c1214c476a44" containerName="istio-proxy"
Sep 18 19:03:46.238942 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.238917    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.241710 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.241685    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ce3141-openshift-default-dockercfg-62fb8\""
Sep 18 19:03:46.253862 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.253834    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"]
Sep 18 19:03:46.358144 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.358111    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmktl\" (UniqueName: \"kubernetes.io/projected/ad321b35-b531-4362-9eeb-9edcecf19c73-kube-api-access-nmktl\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.358144 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.358153    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad321b35-b531-4362-9eeb-9edcecf19c73-istiod-ca-cert\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.358375 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.358295    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-workload-certs\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.358375 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.358361    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-token\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.358444 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.358411    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-envoy\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.358496 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.358481    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-credential-socket\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.358534 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.358504    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-data\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.358575 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.358545    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-workload-socket\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.358611 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.358581    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-podinfo\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.460048 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.459960    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-workload-certs\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.460048 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.460010    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-token\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.460048 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.460035    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-envoy\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.460377 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.460099    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-credential-socket\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.460377 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.460122    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-data\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.460377 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.460149    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-workload-socket\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.460377 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.460172    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-podinfo\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.460377 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.460238    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmktl\" (UniqueName: \"kubernetes.io/projected/ad321b35-b531-4362-9eeb-9edcecf19c73-kube-api-access-nmktl\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.460377 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.460275    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad321b35-b531-4362-9eeb-9edcecf19c73-istiod-ca-cert\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.460685 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.460516    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-credential-socket\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.460685 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.460673    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-data\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.460795 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.460701    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-workload-socket\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.461047 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.461028    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-workload-certs\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.461136 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.461116    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad321b35-b531-4362-9eeb-9edcecf19c73-istiod-ca-cert\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.462514 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.462489    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-envoy\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.462940 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.462918    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-podinfo\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.472565 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.472542    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-token\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.472786 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.472767    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmktl\" (UniqueName: \"kubernetes.io/projected/ad321b35-b531-4362-9eeb-9edcecf19c73-kube-api-access-nmktl\") pod \"e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") " pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.500964 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.500941    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8953-607e3f"]
Sep 18 19:03:46.502685 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:03:46.502661    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98fdb1ec_7158_4908_998a_c31920f58780.slice/crio-02cac6a6a2a8b6752a4add49bdf01691bc0321e6b798496ec9a219db9b150da5 WatchSource:0}: Error finding container 02cac6a6a2a8b6752a4add49bdf01691bc0321e6b798496ec9a219db9b150da5: Status 404 returned error can't find the container with id 02cac6a6a2a8b6752a4add49bdf01691bc0321e6b798496ec9a219db9b150da5
Sep 18 19:03:46.564532 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.564505    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:46.670181 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.670145    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8953-607e3f" event={"ID":"98fdb1ec-7158-4908-998a-c31920f58780","Type":"ContainerStarted","Data":"77122aba352474dbcd92ca916774a58097275e188db328e9835b3b009cf7027d"}
Sep 18 19:03:46.670375 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.670203    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8953-607e3f" event={"ID":"98fdb1ec-7158-4908-998a-c31920f58780","Type":"ContainerStarted","Data":"02cac6a6a2a8b6752a4add49bdf01691bc0321e6b798496ec9a219db9b150da5"}
Sep 18 19:03:46.688640 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.688587    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8953-607e3f" podStartSLOduration=1.688572287 podStartE2EDuration="1.688572287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:03:46.686783674 +0000 UTC m=+1486.123362518" watchObservedRunningTime="2026-09-18 19:03:46.688572287 +0000 UTC m=+1486.125151129"
Sep 18 19:03:46.847668 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.847644    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"]
Sep 18 19:03:46.850206 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:03:46.850163    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad321b35_b531_4362_9eeb_9edcecf19c73.slice/crio-74e8b69f7f703d10971b2c1ada2f48769e5742b5fba66228af1b550c8dd958e8 WatchSource:0}: Error finding container 74e8b69f7f703d10971b2c1ada2f48769e5742b5fba66228af1b550c8dd958e8: Status 404 returned error can't find the container with id 74e8b69f7f703d10971b2c1ada2f48769e5742b5fba66228af1b550c8dd958e8
Sep 18 19:03:46.852724 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.852688    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:03:46.852813 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.852772    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:03:46.852888 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:46.852820    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:03:47.251049 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:47.251014    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8953-607e3f"]
Sep 18 19:03:47.260706 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:47.260668    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8953-607e3f"]
Sep 18 19:03:47.677973 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:47.677933    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k" event={"ID":"ad321b35-b531-4362-9eeb-9edcecf19c73","Type":"ContainerStarted","Data":"2fa01ff0f6f172e485819b3c3dfcae927efe3fef227f597b374a39947a22c8a7"}
Sep 18 19:03:47.678159 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:47.678031    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k" event={"ID":"ad321b35-b531-4362-9eeb-9edcecf19c73","Type":"ContainerStarted","Data":"74e8b69f7f703d10971b2c1ada2f48769e5742b5fba66228af1b550c8dd958e8"}
Sep 18 19:03:47.678303 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:03:47.678287    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8953-607e3f" container="test-curl-8953-607e3f"
Sep 18 19:03:47.699470 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:47.699425    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k" podStartSLOduration=1.699409709 podStartE2EDuration="1.699409709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:03:47.697000168 +0000 UTC m=+1487.133579011" watchObservedRunningTime="2026-09-18 19:03:47.699409709 +0000 UTC m=+1487.135988550"
Sep 18 19:03:48.565263 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:48.565226    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:48.576032 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:48.576008    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:48.684515 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:48.684475    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8953-607e3f" podUID="98fdb1ec-7158-4908-998a-c31920f58780" containerName="test-curl-8953-607e3f" containerID="cri-o://77122aba352474dbcd92ca916774a58097275e188db328e9835b3b009cf7027d" gracePeriod=2
Sep 18 19:03:48.684753 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:48.684720    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:48.685678 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:48.685655    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:03:50.255220 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:50.255168    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"]
Sep 18 19:03:50.865838 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:50.865814    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8953-607e3f"
Sep 18 19:03:51.007880 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:51.007793    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7jmb\" (UniqueName: \"kubernetes.io/projected/98fdb1ec-7158-4908-998a-c31920f58780-kube-api-access-b7jmb\") pod \"98fdb1ec-7158-4908-998a-c31920f58780\" (UID: \"98fdb1ec-7158-4908-998a-c31920f58780\") "
Sep 18 19:03:51.010005 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:51.009968    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fdb1ec-7158-4908-998a-c31920f58780-kube-api-access-b7jmb" (OuterVolumeSpecName: "kube-api-access-b7jmb") pod "98fdb1ec-7158-4908-998a-c31920f58780" (UID: "98fdb1ec-7158-4908-998a-c31920f58780"). InnerVolumeSpecName "kube-api-access-b7jmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:03:51.109040 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:51.108999    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7jmb\" (UniqueName: \"kubernetes.io/projected/98fdb1ec-7158-4908-998a-c31920f58780-kube-api-access-b7jmb\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:03:51.704969 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:51.704929    2558 generic.go:356] "Generic (PLEG): container finished" podID="98fdb1ec-7158-4908-998a-c31920f58780" containerID="77122aba352474dbcd92ca916774a58097275e188db328e9835b3b009cf7027d" exitCode=137
Sep 18 19:03:51.705438 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:51.704973    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8953-607e3f"
Sep 18 19:03:51.705438 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:51.705026    2558 scope.go:117] "RemoveContainer" containerID="77122aba352474dbcd92ca916774a58097275e188db328e9835b3b009cf7027d"
Sep 18 19:03:51.721103 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:51.721077    2558 scope.go:117] "RemoveContainer" containerID="77122aba352474dbcd92ca916774a58097275e188db328e9835b3b009cf7027d"
Sep 18 19:03:51.721247 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:03:51.721214    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8953-607e3f\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="98fdb1ec-7158-4908-998a-c31920f58780" pod="opendatahub/test-curl-8953-607e3f"
Sep 18 19:03:51.721414 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:03:51.721394    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77122aba352474dbcd92ca916774a58097275e188db328e9835b3b009cf7027d\": container with ID starting with 77122aba352474dbcd92ca916774a58097275e188db328e9835b3b009cf7027d not found: ID does not exist" containerID="77122aba352474dbcd92ca916774a58097275e188db328e9835b3b009cf7027d"
Sep 18 19:03:51.721458 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:51.721423    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77122aba352474dbcd92ca916774a58097275e188db328e9835b3b009cf7027d"} err="failed to get container status \"77122aba352474dbcd92ca916774a58097275e188db328e9835b3b009cf7027d\": rpc error: code = NotFound desc = could not find container \"77122aba352474dbcd92ca916774a58097275e188db328e9835b3b009cf7027d\": container with ID starting with 77122aba352474dbcd92ca916774a58097275e188db328e9835b3b009cf7027d not found: ID does not exist"
Sep 18 19:03:52.576150 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:03:52.576105    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8953-607e3f\" is forbidden: User \"system:node:ip-10-0-130-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-78.ec2.internal' and this object" podUID="98fdb1ec-7158-4908-998a-c31920f58780" pod="opendatahub/test-curl-8953-607e3f"
Sep 18 19:03:52.578666 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:03:52.578631    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98fdb1ec-7158-4908-998a-c31920f58780" path="/var/lib/kubelet/pods/98fdb1ec-7158-4908-998a-c31920f58780/volumes"
Sep 18 19:04:02.718602 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:02.718562    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 19:04:02.718602 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:02.718591    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 19:04:02.721706 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:02.721683    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 19:04:02.721894 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:02.721870    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 19:04:05.274749 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.274690    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd" podUID="bdaff6ee-282e-437f-af3b-5bdc4a1e13cd" containerName="main" containerID="cri-o://1ca5be14f73409e850a27032fbcedbb83e45de1d859287a03aac46c5544a7bad" gracePeriod=45
Sep 18 19:04:05.547335 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.547311    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:04:05.648351 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.648319    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-home\") pod \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") "
Sep 18 19:04:05.648550 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.648372    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-tls-certs\") pod \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") "
Sep 18 19:04:05.648550 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.648390    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-model-cache\") pod \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") "
Sep 18 19:04:05.648550 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.648419    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-dshm\") pod \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") "
Sep 18 19:04:05.648550 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.648503    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-tmp-dir\") pod \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") "
Sep 18 19:04:05.648550 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.648548    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxbgf\" (UniqueName: \"kubernetes.io/projected/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-kube-api-access-wxbgf\") pod \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\" (UID: \"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd\") "
Sep 18 19:04:05.648832 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.648781    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdaff6ee-282e-437f-af3b-5bdc4a1e13cd" (UID: "bdaff6ee-282e-437f-af3b-5bdc4a1e13cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:04:05.648889 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.648851    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdaff6ee-282e-437f-af3b-5bdc4a1e13cd" (UID: "bdaff6ee-282e-437f-af3b-5bdc4a1e13cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:04:05.648945 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.648617    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-home" (OuterVolumeSpecName: "home") pod "bdaff6ee-282e-437f-af3b-5bdc4a1e13cd" (UID: "bdaff6ee-282e-437f-af3b-5bdc4a1e13cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:04:05.649000 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.648948    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-tmp-dir\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:04:05.649000 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.648966    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-model-cache\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:04:05.651017 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.650985    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-dshm" (OuterVolumeSpecName: "dshm") pod "bdaff6ee-282e-437f-af3b-5bdc4a1e13cd" (UID: "bdaff6ee-282e-437f-af3b-5bdc4a1e13cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:04:05.651278 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.651245    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdaff6ee-282e-437f-af3b-5bdc4a1e13cd" (UID: "bdaff6ee-282e-437f-af3b-5bdc4a1e13cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:04:05.651374 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.651357    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-kube-api-access-wxbgf" (OuterVolumeSpecName: "kube-api-access-wxbgf") pod "bdaff6ee-282e-437f-af3b-5bdc4a1e13cd" (UID: "bdaff6ee-282e-437f-af3b-5bdc4a1e13cd"). InnerVolumeSpecName "kube-api-access-wxbgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:04:05.750414 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.750369    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-home\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:04:05.750414 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.750405    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-tls-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:04:05.750414 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.750420    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-dshm\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:04:05.750647 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.750432    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxbgf\" (UniqueName: \"kubernetes.io/projected/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd-kube-api-access-wxbgf\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:04:05.805328 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.805240    2558 generic.go:356] "Generic (PLEG): container finished" podID="bdaff6ee-282e-437f-af3b-5bdc4a1e13cd" containerID="1ca5be14f73409e850a27032fbcedbb83e45de1d859287a03aac46c5544a7bad" exitCode=0
Sep 18 19:04:05.805328 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.805281    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"
Sep 18 19:04:05.805494 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.805328    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd" event={"ID":"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd","Type":"ContainerDied","Data":"1ca5be14f73409e850a27032fbcedbb83e45de1d859287a03aac46c5544a7bad"}
Sep 18 19:04:05.805494 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.805373    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd" event={"ID":"bdaff6ee-282e-437f-af3b-5bdc4a1e13cd","Type":"ContainerDied","Data":"ab9e18280eb676b05d34f8d4d0f34a27c94a1e5a883111c9555dd5416edadad3"}
Sep 18 19:04:05.805494 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.805391    2558 scope.go:117] "RemoveContainer" containerID="1ca5be14f73409e850a27032fbcedbb83e45de1d859287a03aac46c5544a7bad"
Sep 18 19:04:05.816833 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.816810    2558 scope.go:117] "RemoveContainer" containerID="1ca5be14f73409e850a27032fbcedbb83e45de1d859287a03aac46c5544a7bad"
Sep 18 19:04:05.817134 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:04:05.817110    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca5be14f73409e850a27032fbcedbb83e45de1d859287a03aac46c5544a7bad\": container with ID starting with 1ca5be14f73409e850a27032fbcedbb83e45de1d859287a03aac46c5544a7bad not found: ID does not exist" containerID="1ca5be14f73409e850a27032fbcedbb83e45de1d859287a03aac46c5544a7bad"
Sep 18 19:04:05.817231 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.817145    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca5be14f73409e850a27032fbcedbb83e45de1d859287a03aac46c5544a7bad"} err="failed to get container status \"1ca5be14f73409e850a27032fbcedbb83e45de1d859287a03aac46c5544a7bad\": rpc error: code = NotFound desc = could not find container \"1ca5be14f73409e850a27032fbcedbb83e45de1d859287a03aac46c5544a7bad\": container with ID starting with 1ca5be14f73409e850a27032fbcedbb83e45de1d859287a03aac46c5544a7bad not found: ID does not exist"
Sep 18 19:04:05.829901 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.829878    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"]
Sep 18 19:04:05.834208 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:05.834172    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/auth-test-model-a6d509-kserve-7849ffc6f9-27dwd"]
Sep 18 19:04:06.575896 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:06.575861    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdaff6ee-282e-437f-af3b-5bdc4a1e13cd" path="/var/lib/kubelet/pods/bdaff6ee-282e-437f-af3b-5bdc4a1e13cd/volumes"
Sep 18 19:04:07.848544 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.848508    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"]
Sep 18 19:04:07.849221 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.849178    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98fdb1ec-7158-4908-998a-c31920f58780" containerName="test-curl-8953-607e3f"
Sep 18 19:04:07.849290 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.849228    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="98fdb1ec-7158-4908-998a-c31920f58780" containerName="test-curl-8953-607e3f"
Sep 18 19:04:07.849290 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.849249    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdaff6ee-282e-437f-af3b-5bdc4a1e13cd" containerName="main"
Sep 18 19:04:07.849290 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.849258    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdaff6ee-282e-437f-af3b-5bdc4a1e13cd" containerName="main"
Sep 18 19:04:07.849441 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.849425    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="98fdb1ec-7158-4908-998a-c31920f58780" containerName="test-curl-8953-607e3f"
Sep 18 19:04:07.849477 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.849446    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdaff6ee-282e-437f-af3b-5bdc4a1e13cd" containerName="main"
Sep 18 19:04:07.854512 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.854475    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.859421 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.859394    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-298125\"/\"default-dockercfg-tg8lq\""
Sep 18 19:04:07.859674 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.859652    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-298125\"/\"sub-test-model-298125-kserve-self-signed-certs\""
Sep 18 19:04:07.859929 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.859903    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-298125\"/\"openshift-service-ca.crt\""
Sep 18 19:04:07.860149 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.860128    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-298125\"/\"kube-root-ca.crt\""
Sep 18 19:04:07.864002 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.863981    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"]
Sep 18 19:04:07.873153 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.873123    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv6bb\" (UniqueName: \"kubernetes.io/projected/aeac8673-9a75-405c-9e16-c522a777a629-kube-api-access-tv6bb\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.873303 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.873171    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-home\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.873303 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.873270    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-model-cache\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.873426 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.873358    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-dshm\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.873426 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.873382    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeac8673-9a75-405c-9e16-c522a777a629-tls-certs\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.873536 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.873448    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-tmp-dir\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.974451 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.974410    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-dshm\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.974451 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.974449    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeac8673-9a75-405c-9e16-c522a777a629-tls-certs\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.974727 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.974500    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-tmp-dir\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.974727 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.974542    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv6bb\" (UniqueName: \"kubernetes.io/projected/aeac8673-9a75-405c-9e16-c522a777a629-kube-api-access-tv6bb\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.974727 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.974588    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-home\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.974727 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.974632    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-model-cache\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.975027 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.975000    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-tmp-dir\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.975124 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.975057    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-home\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.975124 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.975080    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-model-cache\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.976828 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.976801    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-dshm\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.977116 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.977100    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeac8673-9a75-405c-9e16-c522a777a629-tls-certs\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:07.982881 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:07.982858    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv6bb\" (UniqueName: \"kubernetes.io/projected/aeac8673-9a75-405c-9e16-c522a777a629-kube-api-access-tv6bb\") pod \"sub-test-model-298125-kserve-69bd994cf7-xbzsn\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") " pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:08.169846 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:08.169748    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:08.449860 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:08.449829    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"]
Sep 18 19:04:08.451608 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:04:08.451569    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaeac8673_9a75_405c_9e16_c522a777a629.slice/crio-97179a19cc4690022f0bac38cc6fed65b597e0778a268997f3c1a618bb184bd7 WatchSource:0}: Error finding container 97179a19cc4690022f0bac38cc6fed65b597e0778a268997f3c1a618bb184bd7: Status 404 returned error can't find the container with id 97179a19cc4690022f0bac38cc6fed65b597e0778a268997f3c1a618bb184bd7
Sep 18 19:04:08.830850 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:08.830741    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn" event={"ID":"aeac8673-9a75-405c-9e16-c522a777a629","Type":"ContainerStarted","Data":"79cc1bf61bd8c31a3853614e5005c697c64823f28cbc4da806cb505c54970242"}
Sep 18 19:04:08.830850 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:08.830791    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn" event={"ID":"aeac8673-9a75-405c-9e16-c522a777a629","Type":"ContainerStarted","Data":"97179a19cc4690022f0bac38cc6fed65b597e0778a268997f3c1a618bb184bd7"}
Sep 18 19:04:08.832404 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:08.832382    2558 generic.go:356] "Generic (PLEG): container finished" podID="ad321b35-b531-4362-9eeb-9edcecf19c73" containerID="2fa01ff0f6f172e485819b3c3dfcae927efe3fef227f597b374a39947a22c8a7" exitCode=0
Sep 18 19:04:08.832492 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:08.832451    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k" event={"ID":"ad321b35-b531-4362-9eeb-9edcecf19c73","Type":"ContainerDied","Data":"2fa01ff0f6f172e485819b3c3dfcae927efe3fef227f597b374a39947a22c8a7"}
Sep 18 19:04:08.832823 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:08.832808    2558 scope.go:117] "RemoveContainer" containerID="2fa01ff0f6f172e485819b3c3dfcae927efe3fef227f597b374a39947a22c8a7"
Sep 18 19:04:08.833910 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:08.833881    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:04:08.833976 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:08.833949    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:04:08.834014 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:08.833999    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:04:08.850583 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:08.850541    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn" podStartSLOduration=1.850527904 podStartE2EDuration="1.850527904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:04:08.84953476 +0000 UTC m=+1508.286113602" watchObservedRunningTime="2026-09-18 19:04:08.850527904 +0000 UTC m=+1508.287106745"
Sep 18 19:04:09.840677 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:09.840635    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k" event={"ID":"ad321b35-b531-4362-9eeb-9edcecf19c73","Type":"ContainerStarted","Data":"6a6c5f04c26f98febb71309f40a75e7739c9d7e6fd4f8d2d1e819f4929cde5e4"}
Sep 18 19:04:10.565822 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:10.565777    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:04:10.566489 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:10.566456    2558 patch_prober.go:28] interesting pod/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 18 19:04:10.566693 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:10.566663    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k" podUID="ad321b35-b531-4362-9eeb-9edcecf19c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 18 19:04:11.565741 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:11.565695    2558 patch_prober.go:28] interesting pod/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 18 19:04:11.565992 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:11.565778    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k" podUID="ad321b35-b531-4362-9eeb-9edcecf19c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 18 19:04:12.565654 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:12.565614    2558 patch_prober.go:28] interesting pod/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 18 19:04:12.565950 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:12.565920    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k" podUID="ad321b35-b531-4362-9eeb-9edcecf19c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 18 19:04:13.565579 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:13.565538    2558 patch_prober.go:28] interesting pod/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 18 19:04:13.566223 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:13.566163    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k" podUID="ad321b35-b531-4362-9eeb-9edcecf19c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 18 19:04:14.565570 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:14.565528    2558 patch_prober.go:28] interesting pod/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 18 19:04:14.565788 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:14.565605    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k" podUID="ad321b35-b531-4362-9eeb-9edcecf19c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 18 19:04:15.570164 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:15.570094    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:04:15.570575 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:15.570555    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:04:15.571274 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:15.571253    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:04:18.169913 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:18.169882    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:18.170309 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:18.169928    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:18.182928 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:18.182904    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:18.919676 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:18.919641    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:04:20.416245 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.416211    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"]
Sep 18 19:04:20.421148 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.421123    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.423452 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.423432    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-02e2c9-openshift-default-dockercfg-blpgt\""
Sep 18 19:04:20.431745 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.431712    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"]
Sep 18 19:04:20.503609 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.503572    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8f5988-ad2e-472d-bfec-32425b822948-istio-token\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.503798 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.503620    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcn79\" (UniqueName: \"kubernetes.io/projected/1c8f5988-ad2e-472d-bfec-32425b822948-kube-api-access-lcn79\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.503798 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.503663    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-workload-socket\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.503798 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.503684    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-workload-certs\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.503930 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.503831    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-credential-socket\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.503930 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.503875    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-istio-data\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.504009 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.503945    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8f5988-ad2e-472d-bfec-32425b822948-istiod-ca-cert\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.504009 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.503976    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8f5988-ad2e-472d-bfec-32425b822948-istio-podinfo\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.504081 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.504032    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-istio-envoy\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.604606 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.604572    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-credential-socket\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.604606 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.604611    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-istio-data\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.604858 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.604742    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8f5988-ad2e-472d-bfec-32425b822948-istiod-ca-cert\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.604858 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.604789    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8f5988-ad2e-472d-bfec-32425b822948-istio-podinfo\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.604858 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.604845    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-istio-envoy\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.605033 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.604966    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-istio-data\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.605033 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.605005    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8f5988-ad2e-472d-bfec-32425b822948-istio-token\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.605033 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.605023    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-credential-socket\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.605232 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.605051    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcn79\" (UniqueName: \"kubernetes.io/projected/1c8f5988-ad2e-472d-bfec-32425b822948-kube-api-access-lcn79\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.605232 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.605092    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-workload-socket\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.605232 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.605122    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-workload-certs\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.605419 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.605400    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-workload-socket\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.605518 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.605494    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-workload-certs\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.605582 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.605522    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8f5988-ad2e-472d-bfec-32425b822948-istiod-ca-cert\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.607172 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.607151    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-istio-envoy\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.607352 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.607334    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8f5988-ad2e-472d-bfec-32425b822948-istio-podinfo\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.613108 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.613083    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8f5988-ad2e-472d-bfec-32425b822948-istio-token\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.613220 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.613175    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcn79\" (UniqueName: \"kubernetes.io/projected/1c8f5988-ad2e-472d-bfec-32425b822948-kube-api-access-lcn79\") pod \"e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") " pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:20.733681 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:20.733601    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:21.015687 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:21.015662    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"]
Sep 18 19:04:21.017550 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:04:21.017526    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8f5988_ad2e_472d_bfec_32425b822948.slice/crio-87c8ea5f78e6da1f83b840e7e93f118375b862cf352ed22c78dd196b9146c251 WatchSource:0}: Error finding container 87c8ea5f78e6da1f83b840e7e93f118375b862cf352ed22c78dd196b9146c251: Status 404 returned error can't find the container with id 87c8ea5f78e6da1f83b840e7e93f118375b862cf352ed22c78dd196b9146c251
Sep 18 19:04:21.019867 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:21.019833    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:04:21.019962 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:21.019896    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:04:21.019962 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:21.019923    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:04:21.931596 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:21.931557    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5" event={"ID":"1c8f5988-ad2e-472d-bfec-32425b822948","Type":"ContainerStarted","Data":"8f96a8c81f1faef0afc0490400a16b837cc562f5aa544914bc91cd554e722021"}
Sep 18 19:04:21.931967 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:21.931625    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5" event={"ID":"1c8f5988-ad2e-472d-bfec-32425b822948","Type":"ContainerStarted","Data":"87c8ea5f78e6da1f83b840e7e93f118375b862cf352ed22c78dd196b9146c251"}
Sep 18 19:04:21.954688 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:21.954633    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5" podStartSLOduration=1.954617705 podStartE2EDuration="1.954617705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:04:21.95216755 +0000 UTC m=+1521.388746392" watchObservedRunningTime="2026-09-18 19:04:21.954617705 +0000 UTC m=+1521.391196546"
Sep 18 19:04:22.734866 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:22.734832    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:22.740713 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:22.740687    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:22.938503 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:22.938467    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:22.939589 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:22.939564    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:38.144623 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.144590    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"]
Sep 18 19:04:38.152283 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.151521    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.161768 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.161636    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6d509\"/\"default-dockercfg-kh6mz\""
Sep 18 19:04:38.162198 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.162159    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6d509\"/\"sub-test-model-a6d509-kserve-self-signed-certs\""
Sep 18 19:04:38.171174 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.162998    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6d509\"/\"openshift-service-ca.crt\""
Sep 18 19:04:38.172615 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.172595    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6d509\"/\"kube-root-ca.crt\""
Sep 18 19:04:38.187142 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.187108    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"]
Sep 18 19:04:38.190005 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.189982    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-home\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.190096 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.190019    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-tmp-dir\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.190096 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.190060    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwdvg\" (UniqueName: \"kubernetes.io/projected/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-kube-api-access-fwdvg\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.190169 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.190112    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-dshm\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.190169 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.190137    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-model-cache\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.190250 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.190197    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-tls-certs\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.291020 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.290985    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-home\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.291020 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.291020    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-tmp-dir\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.291309 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.291050    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwdvg\" (UniqueName: \"kubernetes.io/projected/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-kube-api-access-fwdvg\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.291309 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.291085    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-dshm\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.291309 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.291104    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-model-cache\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.291309 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.291135    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-tls-certs\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.291593 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.291494    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-tmp-dir\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.291593 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.291512    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-home\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.291709 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.291589    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-model-cache\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.293596 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.293560    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-dshm\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.293699 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.293685    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-tls-certs\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.298862 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.298832    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwdvg\" (UniqueName: \"kubernetes.io/projected/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-kube-api-access-fwdvg\") pod \"sub-test-model-a6d509-kserve-5c9749cdf7-87kbk\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.474424 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.473968    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:38.757436 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:38.757400    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"]
Sep 18 19:04:39.047165 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:39.046522    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk" event={"ID":"6ef6ca8e-552a-4613-9d90-0a0fc560efe0","Type":"ContainerStarted","Data":"6868ba77b5193ff704d508e901751e2f2b028a22cfe0d1ac3f4759b84fbe7004"}
Sep 18 19:04:39.047165 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:39.046579    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk" event={"ID":"6ef6ca8e-552a-4613-9d90-0a0fc560efe0","Type":"ContainerStarted","Data":"762aa5e4fbdb12edf340420a92b6bf3baa374a553ffb9d062dfe304eed7273f5"}
Sep 18 19:04:44.087516 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:44.087482    2558 generic.go:356] "Generic (PLEG): container finished" podID="1c8f5988-ad2e-472d-bfec-32425b822948" containerID="8f96a8c81f1faef0afc0490400a16b837cc562f5aa544914bc91cd554e722021" exitCode=0
Sep 18 19:04:44.087915 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:44.087554    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5" event={"ID":"1c8f5988-ad2e-472d-bfec-32425b822948","Type":"ContainerDied","Data":"8f96a8c81f1faef0afc0490400a16b837cc562f5aa544914bc91cd554e722021"}
Sep 18 19:04:44.088006 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:44.087990    2558 scope.go:117] "RemoveContainer" containerID="8f96a8c81f1faef0afc0490400a16b837cc562f5aa544914bc91cd554e722021"
Sep 18 19:04:44.089373 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:44.089344    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:04:44.089462 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:44.089412    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:04:44.089504 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:44.089465    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:04:44.109069 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:44.109022    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk" podStartSLOduration=6.109007609 podStartE2EDuration="6.109007609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:04:39.064496319 +0000 UTC m=+1538.501075161" watchObservedRunningTime="2026-09-18 19:04:44.109007609 +0000 UTC m=+1543.545586450"
Sep 18 19:04:45.097796 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:45.097756    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5" event={"ID":"1c8f5988-ad2e-472d-bfec-32425b822948","Type":"ContainerStarted","Data":"bb358d3b96fdb5322b4e078665065c937960707ed98fc807abfe7e6e10ddd872"}
Sep 18 19:04:45.734385 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:45.734350    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:45.734646 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:45.734622    2558 patch_prober.go:28] interesting pod/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Sep 18 19:04:45.734696 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:45.734672    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5" podUID="1c8f5988-ad2e-472d-bfec-32425b822948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Sep 18 19:04:46.734591 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:46.734339    2558 patch_prober.go:28] interesting pod/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Sep 18 19:04:46.734591 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:46.734389    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5" podUID="1c8f5988-ad2e-472d-bfec-32425b822948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Sep 18 19:04:47.735041 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:47.735002    2558 patch_prober.go:28] interesting pod/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Sep 18 19:04:47.735458 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:47.735071    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5" podUID="1c8f5988-ad2e-472d-bfec-32425b822948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Sep 18 19:04:48.475856 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:48.475255    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:48.475856 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:48.475299    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:48.488212 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:48.488159    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:48.734352 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:48.734266    2558 patch_prober.go:28] interesting pod/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Sep 18 19:04:48.734352 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:48.734332    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5" podUID="1c8f5988-ad2e-472d-bfec-32425b822948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Sep 18 19:04:49.136356 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:49.136324    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:04:49.734546 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:49.734513    2558 patch_prober.go:28] interesting pod/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Sep 18 19:04:49.734723 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:49.734565    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5" podUID="1c8f5988-ad2e-472d-bfec-32425b822948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Sep 18 19:04:50.738119 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:50.738090    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:50.738540 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:50.738393    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:50.739071 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:50.739055    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:04:57.186856 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:57.186827    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq"]
Sep 18 19:04:57.191502 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:57.191475    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq"
Sep 18 19:04:57.194506 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:57.194475    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:04:57.194750 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:57.194617    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 19:04:57.200027 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:57.199998    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq"]
Sep 18 19:04:57.301108 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:57.301068    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3eb5817-950f-42d9-8fee-630efcc2a1f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq\" (UID: \"f3eb5817-950f-42d9-8fee-630efcc2a1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq"
Sep 18 19:04:57.402491 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:57.402447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3eb5817-950f-42d9-8fee-630efcc2a1f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq\" (UID: \"f3eb5817-950f-42d9-8fee-630efcc2a1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq"
Sep 18 19:04:57.403079 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:57.403058    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3eb5817-950f-42d9-8fee-630efcc2a1f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq\" (UID: \"f3eb5817-950f-42d9-8fee-630efcc2a1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq"
Sep 18 19:04:57.505871 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:57.505782    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq"
Sep 18 19:04:57.792674 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:04:57.792645    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3eb5817_950f_42d9_8fee_630efcc2a1f6.slice/crio-c90b907d0024248ee295cd5c715d4344379ea317a08c6a15fc1d1dde9a645c32 WatchSource:0}: Error finding container c90b907d0024248ee295cd5c715d4344379ea317a08c6a15fc1d1dde9a645c32: Status 404 returned error can't find the container with id c90b907d0024248ee295cd5c715d4344379ea317a08c6a15fc1d1dde9a645c32
Sep 18 19:04:57.794112 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:57.794082    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq"]
Sep 18 19:04:58.187651 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:58.187618    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq" event={"ID":"f3eb5817-950f-42d9-8fee-630efcc2a1f6","Type":"ContainerStarted","Data":"31dca2a099c690f097eb193412e0ee22119aaaf46389b8e5edfcddf8b8f0f3d1"}
Sep 18 19:04:58.188119 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:58.187681    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq" event={"ID":"f3eb5817-950f-42d9-8fee-630efcc2a1f6","Type":"ContainerStarted","Data":"c90b907d0024248ee295cd5c715d4344379ea317a08c6a15fc1d1dde9a645c32"}
Sep 18 19:04:58.202648 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:58.202600    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq" podStartSLOduration=1.202584556 podStartE2EDuration="1.202584556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:04:58.201314232 +0000 UTC m=+1557.637893075" watchObservedRunningTime="2026-09-18 19:04:58.202584556 +0000 UTC m=+1557.639163397"
Sep 18 19:04:59.195679 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:59.195637    2558 generic.go:356] "Generic (PLEG): container finished" podID="f3eb5817-950f-42d9-8fee-630efcc2a1f6" containerID="31dca2a099c690f097eb193412e0ee22119aaaf46389b8e5edfcddf8b8f0f3d1" exitCode=0
Sep 18 19:04:59.196068 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:04:59.195724    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq" event={"ID":"f3eb5817-950f-42d9-8fee-630efcc2a1f6","Type":"ContainerDied","Data":"31dca2a099c690f097eb193412e0ee22119aaaf46389b8e5edfcddf8b8f0f3d1"}
Sep 18 19:05:00.343603 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:00.343578    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq"
Sep 18 19:05:00.433656 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:00.433626    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3eb5817-950f-42d9-8fee-630efcc2a1f6-maas-api-service-ca\") pod \"f3eb5817-950f-42d9-8fee-630efcc2a1f6\" (UID: \"f3eb5817-950f-42d9-8fee-630efcc2a1f6\") "
Sep 18 19:05:00.434013 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:00.433987    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3eb5817-950f-42d9-8fee-630efcc2a1f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3eb5817-950f-42d9-8fee-630efcc2a1f6" (UID: "f3eb5817-950f-42d9-8fee-630efcc2a1f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:05:00.534717 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:00.534635    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3eb5817-950f-42d9-8fee-630efcc2a1f6-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:01.214107 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:01.214073    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq" event={"ID":"f3eb5817-950f-42d9-8fee-630efcc2a1f6","Type":"ContainerDied","Data":"c90b907d0024248ee295cd5c715d4344379ea317a08c6a15fc1d1dde9a645c32"}
Sep 18 19:05:01.214107 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:01.214107    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq"
Sep 18 19:05:01.214345 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:01.214124    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c90b907d0024248ee295cd5c715d4344379ea317a08c6a15fc1d1dde9a645c32"
Sep 18 19:05:02.950818 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:02.950777    2558 scope.go:117] "RemoveContainer" containerID="53b4971c344a8ae10839cc0031e27ed66dcde9e712e55f2d8cbfb74d69cd5d95"
Sep 18 19:05:02.961298 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:02.961277    2558 scope.go:117] "RemoveContainer" containerID="61ac10ab5fb6557d50152f496bcb0021829c6403f49915f2cf893d19fc162871"
Sep 18 19:05:02.969853 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:02.969834    2558 scope.go:117] "RemoveContainer" containerID="71c9d03e387d71c477ddb518fcc880473a55ce6c43f54d1131e57c1a56ffb3fc"
Sep 18 19:05:13.120582 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:13.120546    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq"]
Sep 18 19:05:13.123475 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:13.123446    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce3141-c7mpq"]
Sep 18 19:05:14.575786 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:14.575752    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3eb5817-950f-42d9-8fee-630efcc2a1f6" path="/var/lib/kubelet/pods/f3eb5817-950f-42d9-8fee-630efcc2a1f6/volumes"
Sep 18 19:05:19.688132 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:19.688097    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"]
Sep 18 19:05:19.688572 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:19.688332    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k" podUID="ad321b35-b531-4362-9eeb-9edcecf19c73" containerName="istio-proxy" containerID="cri-o://6a6c5f04c26f98febb71309f40a75e7739c9d7e6fd4f8d2d1e819f4929cde5e4" gracePeriod=30
Sep 18 19:05:20.041057 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:20.040970    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg"]
Sep 18 19:05:20.041420 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:20.041407    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3eb5817-950f-42d9-8fee-630efcc2a1f6" containerName="revoke-keys"
Sep 18 19:05:20.041470 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:20.041422    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3eb5817-950f-42d9-8fee-630efcc2a1f6" containerName="revoke-keys"
Sep 18 19:05:20.041524 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:20.041515    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3eb5817-950f-42d9-8fee-630efcc2a1f6" containerName="revoke-keys"
Sep 18 19:05:20.046524 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:20.046500    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg"
Sep 18 19:05:20.049264 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:20.049239    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 19:05:20.049528 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:20.049508    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:05:20.053390 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:20.053367    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg"]
Sep 18 19:05:20.132625 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:20.132588    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg\" (UID: \"0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg"
Sep 18 19:05:20.234198 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:20.234149    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg\" (UID: \"0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg"
Sep 18 19:05:20.234804 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:20.234783    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg\" (UID: \"0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg"
Sep 18 19:05:20.360206 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:20.360155    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg"
Sep 18 19:05:20.632967 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:20.632941    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg"]
Sep 18 19:05:20.634344 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:05:20.634315    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0776fa_e2fb_46a7_acab_c4aaa7e64ffa.slice/crio-2572eb0728a2fca7e3925408a397b6b46dbdfed7f14036bc4029069c80cc8fae WatchSource:0}: Error finding container 2572eb0728a2fca7e3925408a397b6b46dbdfed7f14036bc4029069c80cc8fae: Status 404 returned error can't find the container with id 2572eb0728a2fca7e3925408a397b6b46dbdfed7f14036bc4029069c80cc8fae
Sep 18 19:05:21.354845 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:21.354805    2558 generic.go:356] "Generic (PLEG): container finished" podID="0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa" containerID="810a108eea442344feee7991b19cbbf26400ba8de0af90024caed98d81e47a2a" exitCode=0
Sep 18 19:05:21.355323 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:21.354846    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg" event={"ID":"0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa","Type":"ContainerDied","Data":"810a108eea442344feee7991b19cbbf26400ba8de0af90024caed98d81e47a2a"}
Sep 18 19:05:21.355323 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:21.354923    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg" event={"ID":"0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa","Type":"ContainerStarted","Data":"2572eb0728a2fca7e3925408a397b6b46dbdfed7f14036bc4029069c80cc8fae"}
Sep 18 19:05:22.505255 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:22.505229    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg"
Sep 18 19:05:22.558567 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:22.558528    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa-maas-api-service-ca\") pod \"0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa\" (UID: \"0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa\") "
Sep 18 19:05:22.558911 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:22.558888    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa" (UID: "0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:05:22.660170 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:22.660082    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:23.369697 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:23.369671    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg"
Sep 18 19:05:23.369882 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:23.369702    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg" event={"ID":"0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa","Type":"ContainerDied","Data":"2572eb0728a2fca7e3925408a397b6b46dbdfed7f14036bc4029069c80cc8fae"}
Sep 18 19:05:23.369882 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:23.369770    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2572eb0728a2fca7e3925408a397b6b46dbdfed7f14036bc4029069c80cc8fae"
Sep 18 19:05:24.955598 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:24.955526    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:05:25.086056 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.086024    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-credential-socket\") pod \"ad321b35-b531-4362-9eeb-9edcecf19c73\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") "
Sep 18 19:05:25.086305 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.086080    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-envoy\") pod \"ad321b35-b531-4362-9eeb-9edcecf19c73\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") "
Sep 18 19:05:25.086305 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.086115    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-workload-socket\") pod \"ad321b35-b531-4362-9eeb-9edcecf19c73\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") "
Sep 18 19:05:25.086305 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.086165    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-podinfo\") pod \"ad321b35-b531-4362-9eeb-9edcecf19c73\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") "
Sep 18 19:05:25.086305 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.086225    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-workload-certs\") pod \"ad321b35-b531-4362-9eeb-9edcecf19c73\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") "
Sep 18 19:05:25.086305 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.086286    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-token\") pod \"ad321b35-b531-4362-9eeb-9edcecf19c73\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") "
Sep 18 19:05:25.086592 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.086315    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad321b35-b531-4362-9eeb-9edcecf19c73-istiod-ca-cert\") pod \"ad321b35-b531-4362-9eeb-9edcecf19c73\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") "
Sep 18 19:05:25.086592 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.086377    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ad321b35-b531-4362-9eeb-9edcecf19c73" (UID: "ad321b35-b531-4362-9eeb-9edcecf19c73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:25.086592 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.086414    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmktl\" (UniqueName: \"kubernetes.io/projected/ad321b35-b531-4362-9eeb-9edcecf19c73-kube-api-access-nmktl\") pod \"ad321b35-b531-4362-9eeb-9edcecf19c73\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") "
Sep 18 19:05:25.086592 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.086443    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-data\") pod \"ad321b35-b531-4362-9eeb-9edcecf19c73\" (UID: \"ad321b35-b531-4362-9eeb-9edcecf19c73\") "
Sep 18 19:05:25.086592 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.086456    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ad321b35-b531-4362-9eeb-9edcecf19c73" (UID: "ad321b35-b531-4362-9eeb-9edcecf19c73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:25.086837 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.086789    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-data" (OuterVolumeSpecName: "istio-data") pod "ad321b35-b531-4362-9eeb-9edcecf19c73" (UID: "ad321b35-b531-4362-9eeb-9edcecf19c73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:25.086837 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.086825    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:25.086925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.086844    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:25.086925 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.086837    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ad321b35-b531-4362-9eeb-9edcecf19c73" (UID: "ad321b35-b531-4362-9eeb-9edcecf19c73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:25.087057 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.087034    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad321b35-b531-4362-9eeb-9edcecf19c73-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ad321b35-b531-4362-9eeb-9edcecf19c73" (UID: "ad321b35-b531-4362-9eeb-9edcecf19c73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:05:25.088547 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.088522    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ad321b35-b531-4362-9eeb-9edcecf19c73" (UID: "ad321b35-b531-4362-9eeb-9edcecf19c73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:25.088750 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.088727    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad321b35-b531-4362-9eeb-9edcecf19c73-kube-api-access-nmktl" (OuterVolumeSpecName: "kube-api-access-nmktl") pod "ad321b35-b531-4362-9eeb-9edcecf19c73" (UID: "ad321b35-b531-4362-9eeb-9edcecf19c73"). InnerVolumeSpecName "kube-api-access-nmktl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:05:25.089017 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.088994    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ad321b35-b531-4362-9eeb-9edcecf19c73" (UID: "ad321b35-b531-4362-9eeb-9edcecf19c73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 19:05:25.089081 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.089001    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-token" (OuterVolumeSpecName: "istio-token") pod "ad321b35-b531-4362-9eeb-9edcecf19c73" (UID: "ad321b35-b531-4362-9eeb-9edcecf19c73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:05:25.187450 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.187417    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:25.187450 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.187445    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:25.187450 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.187454    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:25.187450 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.187462    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:25.187714 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.187470    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad321b35-b531-4362-9eeb-9edcecf19c73-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:25.187714 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.187478    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmktl\" (UniqueName: \"kubernetes.io/projected/ad321b35-b531-4362-9eeb-9edcecf19c73-kube-api-access-nmktl\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:25.187714 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.187487    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad321b35-b531-4362-9eeb-9edcecf19c73-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:25.384749 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.384714    2558 generic.go:356] "Generic (PLEG): container finished" podID="ad321b35-b531-4362-9eeb-9edcecf19c73" containerID="6a6c5f04c26f98febb71309f40a75e7739c9d7e6fd4f8d2d1e819f4929cde5e4" exitCode=0
Sep 18 19:05:25.384916 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.384768    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"
Sep 18 19:05:25.384916 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.384789    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k" event={"ID":"ad321b35-b531-4362-9eeb-9edcecf19c73","Type":"ContainerDied","Data":"6a6c5f04c26f98febb71309f40a75e7739c9d7e6fd4f8d2d1e819f4929cde5e4"}
Sep 18 19:05:25.384916 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.384848    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k" event={"ID":"ad321b35-b531-4362-9eeb-9edcecf19c73","Type":"ContainerDied","Data":"74e8b69f7f703d10971b2c1ada2f48769e5742b5fba66228af1b550c8dd958e8"}
Sep 18 19:05:25.384916 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.384880    2558 scope.go:117] "RemoveContainer" containerID="6a6c5f04c26f98febb71309f40a75e7739c9d7e6fd4f8d2d1e819f4929cde5e4"
Sep 18 19:05:25.394955 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.394937    2558 scope.go:117] "RemoveContainer" containerID="2fa01ff0f6f172e485819b3c3dfcae927efe3fef227f597b374a39947a22c8a7"
Sep 18 19:05:25.403396 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.403380    2558 scope.go:117] "RemoveContainer" containerID="6a6c5f04c26f98febb71309f40a75e7739c9d7e6fd4f8d2d1e819f4929cde5e4"
Sep 18 19:05:25.403672 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:05:25.403645    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6c5f04c26f98febb71309f40a75e7739c9d7e6fd4f8d2d1e819f4929cde5e4\": container with ID starting with 6a6c5f04c26f98febb71309f40a75e7739c9d7e6fd4f8d2d1e819f4929cde5e4 not found: ID does not exist" containerID="6a6c5f04c26f98febb71309f40a75e7739c9d7e6fd4f8d2d1e819f4929cde5e4"
Sep 18 19:05:25.403774 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.403682    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6c5f04c26f98febb71309f40a75e7739c9d7e6fd4f8d2d1e819f4929cde5e4"} err="failed to get container status \"6a6c5f04c26f98febb71309f40a75e7739c9d7e6fd4f8d2d1e819f4929cde5e4\": rpc error: code = NotFound desc = could not find container \"6a6c5f04c26f98febb71309f40a75e7739c9d7e6fd4f8d2d1e819f4929cde5e4\": container with ID starting with 6a6c5f04c26f98febb71309f40a75e7739c9d7e6fd4f8d2d1e819f4929cde5e4 not found: ID does not exist"
Sep 18 19:05:25.403774 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.403704    2558 scope.go:117] "RemoveContainer" containerID="2fa01ff0f6f172e485819b3c3dfcae927efe3fef227f597b374a39947a22c8a7"
Sep 18 19:05:25.404117 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:05:25.404095    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa01ff0f6f172e485819b3c3dfcae927efe3fef227f597b374a39947a22c8a7\": container with ID starting with 2fa01ff0f6f172e485819b3c3dfcae927efe3fef227f597b374a39947a22c8a7 not found: ID does not exist" containerID="2fa01ff0f6f172e485819b3c3dfcae927efe3fef227f597b374a39947a22c8a7"
Sep 18 19:05:25.404224 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.404125    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa01ff0f6f172e485819b3c3dfcae927efe3fef227f597b374a39947a22c8a7"} err="failed to get container status \"2fa01ff0f6f172e485819b3c3dfcae927efe3fef227f597b374a39947a22c8a7\": rpc error: code = NotFound desc = could not find container \"2fa01ff0f6f172e485819b3c3dfcae927efe3fef227f597b374a39947a22c8a7\": container with ID starting with 2fa01ff0f6f172e485819b3c3dfcae927efe3fef227f597b374a39947a22c8a7 not found: ID does not exist"
Sep 18 19:05:25.407335 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.407314    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"]
Sep 18 19:05:25.412573 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:25.412550    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ce3141-openshift-default-b7764b4cd-sxq2k"]
Sep 18 19:05:26.575960 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:26.575929    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad321b35-b531-4362-9eeb-9edcecf19c73" path="/var/lib/kubelet/pods/ad321b35-b531-4362-9eeb-9edcecf19c73/volumes"
Sep 18 19:05:29.446765 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:29.446720    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"]
Sep 18 19:05:30.359733 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:30.359695    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"]
Sep 18 19:05:33.652545 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.652510    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"]
Sep 18 19:05:33.653134 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.653117    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad321b35-b531-4362-9eeb-9edcecf19c73" containerName="istio-proxy"
Sep 18 19:05:33.653239 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.653136    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad321b35-b531-4362-9eeb-9edcecf19c73" containerName="istio-proxy"
Sep 18 19:05:33.653239 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.653163    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa" containerName="revoke-keys"
Sep 18 19:05:33.653239 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.653172    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa" containerName="revoke-keys"
Sep 18 19:05:33.653239 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.653216    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad321b35-b531-4362-9eeb-9edcecf19c73" containerName="istio-proxy"
Sep 18 19:05:33.653239 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.653225    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad321b35-b531-4362-9eeb-9edcecf19c73" containerName="istio-proxy"
Sep 18 19:05:33.653502 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.653346    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad321b35-b531-4362-9eeb-9edcecf19c73" containerName="istio-proxy"
Sep 18 19:05:33.653502 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.653362    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa" containerName="revoke-keys"
Sep 18 19:05:33.658641 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.658615    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.661347 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.661326    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-298125\"/\"rate-test-model-298125-kserve-self-signed-certs\""
Sep 18 19:05:33.671064 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.671043    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"]
Sep 18 19:05:33.773466 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.773432    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-tmp-dir\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.773643 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.773488    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-home\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.773643 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.773554    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-dshm\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.773643 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.773603    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvprx\" (UniqueName: \"kubernetes.io/projected/71bdd860-9a46-4cac-ad86-fa7dec06f902-kube-api-access-zvprx\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.773643 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.773635    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-model-cache\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.773854 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.773746    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71bdd860-9a46-4cac-ad86-fa7dec06f902-tls-certs\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.874777 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.874726    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-tmp-dir\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.874961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.874815    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-home\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.874961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.874852    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-dshm\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.874961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.874902    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvprx\" (UniqueName: \"kubernetes.io/projected/71bdd860-9a46-4cac-ad86-fa7dec06f902-kube-api-access-zvprx\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.874961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.874947    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-model-cache\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.875117 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.875017    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71bdd860-9a46-4cac-ad86-fa7dec06f902-tls-certs\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.875251 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.875231    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-tmp-dir\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.875307 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.875267    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-home\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.875422 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.875394    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-model-cache\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.877286 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.877267    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-dshm\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.877407 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.877391    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71bdd860-9a46-4cac-ad86-fa7dec06f902-tls-certs\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.883224 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.883204    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvprx\" (UniqueName: \"kubernetes.io/projected/71bdd860-9a46-4cac-ad86-fa7dec06f902-kube-api-access-zvprx\") pod \"rate-test-model-298125-kserve-66557f479-fsrg7\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") " pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:33.968872 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:33.968788    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:34.243382 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:34.243353    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"]
Sep 18 19:05:34.244139 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:05:34.244111    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71bdd860_9a46_4cac_ad86_fa7dec06f902.slice/crio-766c2a263d84484607373bf3e7b7e1b28dd5fefee66398549806a261d4453852 WatchSource:0}: Error finding container 766c2a263d84484607373bf3e7b7e1b28dd5fefee66398549806a261d4453852: Status 404 returned error can't find the container with id 766c2a263d84484607373bf3e7b7e1b28dd5fefee66398549806a261d4453852
Sep 18 19:05:34.444908 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:34.444866    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7" event={"ID":"71bdd860-9a46-4cac-ad86-fa7dec06f902","Type":"ContainerStarted","Data":"62e332cc380698c312d723a21e75fd6291e77e40a1194ff00f4a407eedf5a70a"}
Sep 18 19:05:34.445065 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:34.444918    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7" event={"ID":"71bdd860-9a46-4cac-ad86-fa7dec06f902","Type":"ContainerStarted","Data":"766c2a263d84484607373bf3e7b7e1b28dd5fefee66398549806a261d4453852"}
Sep 18 19:05:34.465873 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:34.465817    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7" podStartSLOduration=1.465800611 podStartE2EDuration="1.465800611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:05:34.464960672 +0000 UTC m=+1593.901539513" watchObservedRunningTime="2026-09-18 19:05:34.465800611 +0000 UTC m=+1593.902379497"
Sep 18 19:05:35.937426 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:35.937388    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg"]
Sep 18 19:05:35.940780 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:35.940753    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-02e2c9-z9tdg"]
Sep 18 19:05:36.576463 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:36.576429    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa" path="/var/lib/kubelet/pods/0e0776fa-e2fb-46a7-acab-c4aaa7e64ffa/volumes"
Sep 18 19:05:42.458276 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:42.458235    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"]
Sep 18 19:05:42.458753 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:42.458468    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5" podUID="1c8f5988-ad2e-472d-bfec-32425b822948" containerName="istio-proxy" containerID="cri-o://bb358d3b96fdb5322b4e078665065c937960707ed98fc807abfe7e6e10ddd872" gracePeriod=30
Sep 18 19:05:42.842213 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:42.842164    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24"]
Sep 18 19:05:42.842684 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:42.842670    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad321b35-b531-4362-9eeb-9edcecf19c73" containerName="istio-proxy"
Sep 18 19:05:42.846021 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:42.846005    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24"
Sep 18 19:05:42.848291 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:42.848272    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:05:42.848413 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:42.848351    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 19:05:42.853523 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:42.853501    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24"]
Sep 18 19:05:42.970724 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:42.970688    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fadacc-d28f-4cb1-976a-c908648db2c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24\" (UID: \"33fadacc-d28f-4cb1-976a-c908648db2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24"
Sep 18 19:05:43.071693 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:43.071655    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fadacc-d28f-4cb1-976a-c908648db2c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24\" (UID: \"33fadacc-d28f-4cb1-976a-c908648db2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24"
Sep 18 19:05:43.072310 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:43.072291    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fadacc-d28f-4cb1-976a-c908648db2c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24\" (UID: \"33fadacc-d28f-4cb1-976a-c908648db2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24"
Sep 18 19:05:43.159095 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:43.159010    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24"
Sep 18 19:05:43.439276 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:43.439245    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24"]
Sep 18 19:05:43.440772 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:05:43.440743    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33fadacc_d28f_4cb1_976a_c908648db2c8.slice/crio-8b8c334e612da5f377b79a68ff4d6b0eb5ed31c331064837b9257dbc2bf8a195 WatchSource:0}: Error finding container 8b8c334e612da5f377b79a68ff4d6b0eb5ed31c331064837b9257dbc2bf8a195: Status 404 returned error can't find the container with id 8b8c334e612da5f377b79a68ff4d6b0eb5ed31c331064837b9257dbc2bf8a195
Sep 18 19:05:43.507614 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:43.507585    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24" event={"ID":"33fadacc-d28f-4cb1-976a-c908648db2c8","Type":"ContainerStarted","Data":"8b8c334e612da5f377b79a68ff4d6b0eb5ed31c331064837b9257dbc2bf8a195"}
Sep 18 19:05:43.968888 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:43.968854    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:43.969055 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:43.968906    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:43.981516 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:43.981487    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:44.466681 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.466590    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn" podUID="aeac8673-9a75-405c-9e16-c522a777a629" containerName="main" containerID="cri-o://79cc1bf61bd8c31a3853614e5005c697c64823f28cbc4da806cb505c54970242" gracePeriod=45
Sep 18 19:05:44.516589 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.516559    2558 generic.go:356] "Generic (PLEG): container finished" podID="33fadacc-d28f-4cb1-976a-c908648db2c8" containerID="6e6bc8f92a39f6a8ac9982bab381ae71305ba50b94fae52a7110df70a050949a" exitCode=0
Sep 18 19:05:44.516920 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.516647    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24" event={"ID":"33fadacc-d28f-4cb1-976a-c908648db2c8","Type":"ContainerDied","Data":"6e6bc8f92a39f6a8ac9982bab381ae71305ba50b94fae52a7110df70a050949a"}
Sep 18 19:05:44.532619 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.532590    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:05:44.733626 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.733568    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:05:44.891593 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.891561    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv6bb\" (UniqueName: \"kubernetes.io/projected/aeac8673-9a75-405c-9e16-c522a777a629-kube-api-access-tv6bb\") pod \"aeac8673-9a75-405c-9e16-c522a777a629\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") "
Sep 18 19:05:44.891593 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.891606    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-tmp-dir\") pod \"aeac8673-9a75-405c-9e16-c522a777a629\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") "
Sep 18 19:05:44.891845 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.891701    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeac8673-9a75-405c-9e16-c522a777a629-tls-certs\") pod \"aeac8673-9a75-405c-9e16-c522a777a629\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") "
Sep 18 19:05:44.891845 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.891727    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-model-cache\") pod \"aeac8673-9a75-405c-9e16-c522a777a629\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") "
Sep 18 19:05:44.891845 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.891753    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-home\") pod \"aeac8673-9a75-405c-9e16-c522a777a629\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") "
Sep 18 19:05:44.891845 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.891785    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-dshm\") pod \"aeac8673-9a75-405c-9e16-c522a777a629\" (UID: \"aeac8673-9a75-405c-9e16-c522a777a629\") "
Sep 18 19:05:44.892040 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.891932    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aeac8673-9a75-405c-9e16-c522a777a629" (UID: "aeac8673-9a75-405c-9e16-c522a777a629"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:44.892040 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.891987    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-model-cache" (OuterVolumeSpecName: "model-cache") pod "aeac8673-9a75-405c-9e16-c522a777a629" (UID: "aeac8673-9a75-405c-9e16-c522a777a629"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:44.892155 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.892073    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-home" (OuterVolumeSpecName: "home") pod "aeac8673-9a75-405c-9e16-c522a777a629" (UID: "aeac8673-9a75-405c-9e16-c522a777a629"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:44.892246 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.892154    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-model-cache\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:44.892246 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.892169    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-home\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:44.892246 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.892206    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-tmp-dir\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:44.894000 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.893968    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeac8673-9a75-405c-9e16-c522a777a629-kube-api-access-tv6bb" (OuterVolumeSpecName: "kube-api-access-tv6bb") pod "aeac8673-9a75-405c-9e16-c522a777a629" (UID: "aeac8673-9a75-405c-9e16-c522a777a629"). InnerVolumeSpecName "kube-api-access-tv6bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:05:44.894114 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.893976    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeac8673-9a75-405c-9e16-c522a777a629-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aeac8673-9a75-405c-9e16-c522a777a629" (UID: "aeac8673-9a75-405c-9e16-c522a777a629"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:05:44.894114 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.894026    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-dshm" (OuterVolumeSpecName: "dshm") pod "aeac8673-9a75-405c-9e16-c522a777a629" (UID: "aeac8673-9a75-405c-9e16-c522a777a629"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:44.993002 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.992907    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeac8673-9a75-405c-9e16-c522a777a629-tls-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:44.993002 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.992942    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeac8673-9a75-405c-9e16-c522a777a629-dshm\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:44.993002 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:44.992956    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv6bb\" (UniqueName: \"kubernetes.io/projected/aeac8673-9a75-405c-9e16-c522a777a629-kube-api-access-tv6bb\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:45.379253 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.379167    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk" podUID="6ef6ca8e-552a-4613-9d90-0a0fc560efe0" containerName="main" containerID="cri-o://6868ba77b5193ff704d508e901751e2f2b028a22cfe0d1ac3f4759b84fbe7004" gracePeriod=45
Sep 18 19:05:45.525625 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.525593    2558 generic.go:356] "Generic (PLEG): container finished" podID="6ef6ca8e-552a-4613-9d90-0a0fc560efe0" containerID="6868ba77b5193ff704d508e901751e2f2b028a22cfe0d1ac3f4759b84fbe7004" exitCode=0
Sep 18 19:05:45.526069 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.525664    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk" event={"ID":"6ef6ca8e-552a-4613-9d90-0a0fc560efe0","Type":"ContainerDied","Data":"6868ba77b5193ff704d508e901751e2f2b028a22cfe0d1ac3f4759b84fbe7004"}
Sep 18 19:05:45.527266 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.527241    2558 generic.go:356] "Generic (PLEG): container finished" podID="aeac8673-9a75-405c-9e16-c522a777a629" containerID="79cc1bf61bd8c31a3853614e5005c697c64823f28cbc4da806cb505c54970242" exitCode=0
Sep 18 19:05:45.527390 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.527282    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"
Sep 18 19:05:45.527390 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.527319    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn" event={"ID":"aeac8673-9a75-405c-9e16-c522a777a629","Type":"ContainerDied","Data":"79cc1bf61bd8c31a3853614e5005c697c64823f28cbc4da806cb505c54970242"}
Sep 18 19:05:45.527390 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.527370    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn" event={"ID":"aeac8673-9a75-405c-9e16-c522a777a629","Type":"ContainerDied","Data":"97179a19cc4690022f0bac38cc6fed65b597e0778a268997f3c1a618bb184bd7"}
Sep 18 19:05:45.527513 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.527394    2558 scope.go:117] "RemoveContainer" containerID="79cc1bf61bd8c31a3853614e5005c697c64823f28cbc4da806cb505c54970242"
Sep 18 19:05:45.540917 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.540895    2558 scope.go:117] "RemoveContainer" containerID="79cc1bf61bd8c31a3853614e5005c697c64823f28cbc4da806cb505c54970242"
Sep 18 19:05:45.541266 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:05:45.541230    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79cc1bf61bd8c31a3853614e5005c697c64823f28cbc4da806cb505c54970242\": container with ID starting with 79cc1bf61bd8c31a3853614e5005c697c64823f28cbc4da806cb505c54970242 not found: ID does not exist" containerID="79cc1bf61bd8c31a3853614e5005c697c64823f28cbc4da806cb505c54970242"
Sep 18 19:05:45.541366 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.541267    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79cc1bf61bd8c31a3853614e5005c697c64823f28cbc4da806cb505c54970242"} err="failed to get container status \"79cc1bf61bd8c31a3853614e5005c697c64823f28cbc4da806cb505c54970242\": rpc error: code = NotFound desc = could not find container \"79cc1bf61bd8c31a3853614e5005c697c64823f28cbc4da806cb505c54970242\": container with ID starting with 79cc1bf61bd8c31a3853614e5005c697c64823f28cbc4da806cb505c54970242 not found: ID does not exist"
Sep 18 19:05:45.558440 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.558369    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"]
Sep 18 19:05:45.562910 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.562882    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-298125/sub-test-model-298125-kserve-69bd994cf7-xbzsn"]
Sep 18 19:05:45.643484 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.643421    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:05:45.690148 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.690120    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24"
Sep 18 19:05:45.698395 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.698373    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-dshm\") pod \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") "
Sep 18 19:05:45.698529 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.698414    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-tmp-dir\") pod \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") "
Sep 18 19:05:45.698529 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.698447    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-model-cache\") pod \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") "
Sep 18 19:05:45.698529 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.698475    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwdvg\" (UniqueName: \"kubernetes.io/projected/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-kube-api-access-fwdvg\") pod \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") "
Sep 18 19:05:45.698529 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.698513    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fadacc-d28f-4cb1-976a-c908648db2c8-maas-api-service-ca\") pod \"33fadacc-d28f-4cb1-976a-c908648db2c8\" (UID: \"33fadacc-d28f-4cb1-976a-c908648db2c8\") "
Sep 18 19:05:45.698760 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.698545    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-tls-certs\") pod \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") "
Sep 18 19:05:45.698760 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.698574    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-home\") pod \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\" (UID: \"6ef6ca8e-552a-4613-9d90-0a0fc560efe0\") "
Sep 18 19:05:45.698760 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.698715    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ef6ca8e-552a-4613-9d90-0a0fc560efe0" (UID: "6ef6ca8e-552a-4613-9d90-0a0fc560efe0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:45.698760 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.698702    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ef6ca8e-552a-4613-9d90-0a0fc560efe0" (UID: "6ef6ca8e-552a-4613-9d90-0a0fc560efe0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:45.698974 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.698799    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-tmp-dir\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:45.698974 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.698815    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-model-cache\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:45.698974 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.698934    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-home" (OuterVolumeSpecName: "home") pod "6ef6ca8e-552a-4613-9d90-0a0fc560efe0" (UID: "6ef6ca8e-552a-4613-9d90-0a0fc560efe0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:45.699106 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.698969    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33fadacc-d28f-4cb1-976a-c908648db2c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33fadacc-d28f-4cb1-976a-c908648db2c8" (UID: "33fadacc-d28f-4cb1-976a-c908648db2c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:05:45.700550 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.700530    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-dshm" (OuterVolumeSpecName: "dshm") pod "6ef6ca8e-552a-4613-9d90-0a0fc560efe0" (UID: "6ef6ca8e-552a-4613-9d90-0a0fc560efe0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:45.700661 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.700640    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-kube-api-access-fwdvg" (OuterVolumeSpecName: "kube-api-access-fwdvg") pod "6ef6ca8e-552a-4613-9d90-0a0fc560efe0" (UID: "6ef6ca8e-552a-4613-9d90-0a0fc560efe0"). InnerVolumeSpecName "kube-api-access-fwdvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:05:45.700704 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.700672    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ef6ca8e-552a-4613-9d90-0a0fc560efe0" (UID: "6ef6ca8e-552a-4613-9d90-0a0fc560efe0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:05:45.799399 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.799362    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-tls-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:45.799399 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.799391    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-home\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:45.799399 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.799399    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-dshm\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:45.799399 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.799407    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwdvg\" (UniqueName: \"kubernetes.io/projected/6ef6ca8e-552a-4613-9d90-0a0fc560efe0-kube-api-access-fwdvg\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:45.799678 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:45.799417    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fadacc-d28f-4cb1-976a-c908648db2c8-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:46.536297 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:46.536264    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24"
Sep 18 19:05:46.536751 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:46.536307    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24" event={"ID":"33fadacc-d28f-4cb1-976a-c908648db2c8","Type":"ContainerDied","Data":"8b8c334e612da5f377b79a68ff4d6b0eb5ed31c331064837b9257dbc2bf8a195"}
Sep 18 19:05:46.536751 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:46.536369    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b8c334e612da5f377b79a68ff4d6b0eb5ed31c331064837b9257dbc2bf8a195"
Sep 18 19:05:46.537810 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:46.537781    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"
Sep 18 19:05:46.537937 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:46.537827    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk" event={"ID":"6ef6ca8e-552a-4613-9d90-0a0fc560efe0","Type":"ContainerDied","Data":"762aa5e4fbdb12edf340420a92b6bf3baa374a553ffb9d062dfe304eed7273f5"}
Sep 18 19:05:46.537937 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:46.537868    2558 scope.go:117] "RemoveContainer" containerID="6868ba77b5193ff704d508e901751e2f2b028a22cfe0d1ac3f4759b84fbe7004"
Sep 18 19:05:46.565894 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:46.565859    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"]
Sep 18 19:05:46.568404 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:46.568383    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/sub-test-model-a6d509-kserve-5c9749cdf7-87kbk"]
Sep 18 19:05:46.576020 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:46.575997    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ef6ca8e-552a-4613-9d90-0a0fc560efe0" path="/var/lib/kubelet/pods/6ef6ca8e-552a-4613-9d90-0a0fc560efe0/volumes"
Sep 18 19:05:46.576450 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:46.576437    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeac8673-9a75-405c-9e16-c522a777a629" path="/var/lib/kubelet/pods/aeac8673-9a75-405c-9e16-c522a777a629/volumes"
Sep 18 19:05:47.731429 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.731372    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:05:47.813272 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.813234    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcn79\" (UniqueName: \"kubernetes.io/projected/1c8f5988-ad2e-472d-bfec-32425b822948-kube-api-access-lcn79\") pod \"1c8f5988-ad2e-472d-bfec-32425b822948\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") "
Sep 18 19:05:47.813272 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.813274    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8f5988-ad2e-472d-bfec-32425b822948-istio-podinfo\") pod \"1c8f5988-ad2e-472d-bfec-32425b822948\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") "
Sep 18 19:05:47.813533 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.813306    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-credential-socket\") pod \"1c8f5988-ad2e-472d-bfec-32425b822948\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") "
Sep 18 19:05:47.813533 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.813338    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-workload-socket\") pod \"1c8f5988-ad2e-472d-bfec-32425b822948\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") "
Sep 18 19:05:47.813533 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.813387    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-istio-envoy\") pod \"1c8f5988-ad2e-472d-bfec-32425b822948\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") "
Sep 18 19:05:47.813533 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.813446    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8f5988-ad2e-472d-bfec-32425b822948-istio-token\") pod \"1c8f5988-ad2e-472d-bfec-32425b822948\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") "
Sep 18 19:05:47.813533 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.813468    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-workload-certs\") pod \"1c8f5988-ad2e-472d-bfec-32425b822948\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") "
Sep 18 19:05:47.813533 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.813494    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-istio-data\") pod \"1c8f5988-ad2e-472d-bfec-32425b822948\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") "
Sep 18 19:05:47.813533 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.813522    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8f5988-ad2e-472d-bfec-32425b822948-istiod-ca-cert\") pod \"1c8f5988-ad2e-472d-bfec-32425b822948\" (UID: \"1c8f5988-ad2e-472d-bfec-32425b822948\") "
Sep 18 19:05:47.813880 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.813652    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1c8f5988-ad2e-472d-bfec-32425b822948" (UID: "1c8f5988-ad2e-472d-bfec-32425b822948"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:47.813944 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.813897    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1c8f5988-ad2e-472d-bfec-32425b822948" (UID: "1c8f5988-ad2e-472d-bfec-32425b822948"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:47.813944 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.813878    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1c8f5988-ad2e-472d-bfec-32425b822948" (UID: "1c8f5988-ad2e-472d-bfec-32425b822948"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:47.814072 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.814053    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:47.814126 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.814094    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:47.814126 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.814065    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-istio-data" (OuterVolumeSpecName: "istio-data") pod "1c8f5988-ad2e-472d-bfec-32425b822948" (UID: "1c8f5988-ad2e-472d-bfec-32425b822948"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:47.814227 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.814093    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8f5988-ad2e-472d-bfec-32425b822948-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1c8f5988-ad2e-472d-bfec-32425b822948" (UID: "1c8f5988-ad2e-472d-bfec-32425b822948"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:05:47.815586 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.815563    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1c8f5988-ad2e-472d-bfec-32425b822948" (UID: "1c8f5988-ad2e-472d-bfec-32425b822948"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:47.815818 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.815794    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8f5988-ad2e-472d-bfec-32425b822948-istio-token" (OuterVolumeSpecName: "istio-token") pod "1c8f5988-ad2e-472d-bfec-32425b822948" (UID: "1c8f5988-ad2e-472d-bfec-32425b822948"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:05:47.816006 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.815977    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c8f5988-ad2e-472d-bfec-32425b822948-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1c8f5988-ad2e-472d-bfec-32425b822948" (UID: "1c8f5988-ad2e-472d-bfec-32425b822948"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 19:05:47.816006 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.815985    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8f5988-ad2e-472d-bfec-32425b822948-kube-api-access-lcn79" (OuterVolumeSpecName: "kube-api-access-lcn79") pod "1c8f5988-ad2e-472d-bfec-32425b822948" (UID: "1c8f5988-ad2e-472d-bfec-32425b822948"). InnerVolumeSpecName "kube-api-access-lcn79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:05:47.914627 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.914593    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8f5988-ad2e-472d-bfec-32425b822948-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:47.914627 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.914619    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:47.914627 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.914628    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:47.914627 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.914636    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8f5988-ad2e-472d-bfec-32425b822948-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:47.914889 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.914645    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcn79\" (UniqueName: \"kubernetes.io/projected/1c8f5988-ad2e-472d-bfec-32425b822948-kube-api-access-lcn79\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:47.914889 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.914655    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8f5988-ad2e-472d-bfec-32425b822948-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:47.914889 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:47.914663    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8f5988-ad2e-472d-bfec-32425b822948-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:05:48.556197 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.556145    2558 generic.go:356] "Generic (PLEG): container finished" podID="1c8f5988-ad2e-472d-bfec-32425b822948" containerID="bb358d3b96fdb5322b4e078665065c937960707ed98fc807abfe7e6e10ddd872" exitCode=0
Sep 18 19:05:48.556352 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.556236    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"
Sep 18 19:05:48.556352 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.556240    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5" event={"ID":"1c8f5988-ad2e-472d-bfec-32425b822948","Type":"ContainerDied","Data":"bb358d3b96fdb5322b4e078665065c937960707ed98fc807abfe7e6e10ddd872"}
Sep 18 19:05:48.556456 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.556348    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5" event={"ID":"1c8f5988-ad2e-472d-bfec-32425b822948","Type":"ContainerDied","Data":"87c8ea5f78e6da1f83b840e7e93f118375b862cf352ed22c78dd196b9146c251"}
Sep 18 19:05:48.556456 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.556383    2558 scope.go:117] "RemoveContainer" containerID="bb358d3b96fdb5322b4e078665065c937960707ed98fc807abfe7e6e10ddd872"
Sep 18 19:05:48.567021 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.567005    2558 scope.go:117] "RemoveContainer" containerID="8f96a8c81f1faef0afc0490400a16b837cc562f5aa544914bc91cd554e722021"
Sep 18 19:05:48.576570 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.575981    2558 scope.go:117] "RemoveContainer" containerID="bb358d3b96fdb5322b4e078665065c937960707ed98fc807abfe7e6e10ddd872"
Sep 18 19:05:48.576570 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:05:48.576317    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb358d3b96fdb5322b4e078665065c937960707ed98fc807abfe7e6e10ddd872\": container with ID starting with bb358d3b96fdb5322b4e078665065c937960707ed98fc807abfe7e6e10ddd872 not found: ID does not exist" containerID="bb358d3b96fdb5322b4e078665065c937960707ed98fc807abfe7e6e10ddd872"
Sep 18 19:05:48.576570 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.576351    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb358d3b96fdb5322b4e078665065c937960707ed98fc807abfe7e6e10ddd872"} err="failed to get container status \"bb358d3b96fdb5322b4e078665065c937960707ed98fc807abfe7e6e10ddd872\": rpc error: code = NotFound desc = could not find container \"bb358d3b96fdb5322b4e078665065c937960707ed98fc807abfe7e6e10ddd872\": container with ID starting with bb358d3b96fdb5322b4e078665065c937960707ed98fc807abfe7e6e10ddd872 not found: ID does not exist"
Sep 18 19:05:48.576570 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.576375    2558 scope.go:117] "RemoveContainer" containerID="8f96a8c81f1faef0afc0490400a16b837cc562f5aa544914bc91cd554e722021"
Sep 18 19:05:48.577094 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:05:48.577061    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f96a8c81f1faef0afc0490400a16b837cc562f5aa544914bc91cd554e722021\": container with ID starting with 8f96a8c81f1faef0afc0490400a16b837cc562f5aa544914bc91cd554e722021 not found: ID does not exist" containerID="8f96a8c81f1faef0afc0490400a16b837cc562f5aa544914bc91cd554e722021"
Sep 18 19:05:48.577178 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.577100    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f96a8c81f1faef0afc0490400a16b837cc562f5aa544914bc91cd554e722021"} err="failed to get container status \"8f96a8c81f1faef0afc0490400a16b837cc562f5aa544914bc91cd554e722021\": rpc error: code = NotFound desc = could not find container \"8f96a8c81f1faef0afc0490400a16b837cc562f5aa544914bc91cd554e722021\": container with ID starting with 8f96a8c81f1faef0afc0490400a16b837cc562f5aa544914bc91cd554e722021 not found: ID does not exist"
Sep 18 19:05:48.580453 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.580426    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"]
Sep 18 19:05:48.583284 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.583265    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-02e2c9-openshift-default-774f47998d-jm9w5"]
Sep 18 19:05:48.861021 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.860988    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"]
Sep 18 19:05:48.861408 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.861397    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c8f5988-ad2e-472d-bfec-32425b822948" containerName="istio-proxy"
Sep 18 19:05:48.861408 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.861408    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c8f5988-ad2e-472d-bfec-32425b822948" containerName="istio-proxy"
Sep 18 19:05:48.861481 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.861418    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ef6ca8e-552a-4613-9d90-0a0fc560efe0" containerName="main"
Sep 18 19:05:48.861481 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.861424    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ef6ca8e-552a-4613-9d90-0a0fc560efe0" containerName="main"
Sep 18 19:05:48.861481 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.861436    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33fadacc-d28f-4cb1-976a-c908648db2c8" containerName="revoke-keys"
Sep 18 19:05:48.861481 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.861442    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="33fadacc-d28f-4cb1-976a-c908648db2c8" containerName="revoke-keys"
Sep 18 19:05:48.861481 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.861452    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c8f5988-ad2e-472d-bfec-32425b822948" containerName="istio-proxy"
Sep 18 19:05:48.861481 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.861457    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c8f5988-ad2e-472d-bfec-32425b822948" containerName="istio-proxy"
Sep 18 19:05:48.861481 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.861467    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeac8673-9a75-405c-9e16-c522a777a629" containerName="main"
Sep 18 19:05:48.861481 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.861472    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeac8673-9a75-405c-9e16-c522a777a629" containerName="main"
Sep 18 19:05:48.861716 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.861535    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeac8673-9a75-405c-9e16-c522a777a629" containerName="main"
Sep 18 19:05:48.861716 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.861542    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="33fadacc-d28f-4cb1-976a-c908648db2c8" containerName="revoke-keys"
Sep 18 19:05:48.861716 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.861549    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c8f5988-ad2e-472d-bfec-32425b822948" containerName="istio-proxy"
Sep 18 19:05:48.861716 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.861555    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c8f5988-ad2e-472d-bfec-32425b822948" containerName="istio-proxy"
Sep 18 19:05:48.861716 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.861560    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ef6ca8e-552a-4613-9d90-0a0fc560efe0" containerName="main"
Sep 18 19:05:48.866172 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.866153    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:48.869901 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.869880    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6d509\"/\"kube-root-ca.crt\""
Sep 18 19:05:48.870081 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.869956    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6d509\"/\"openshift-service-ca.crt\""
Sep 18 19:05:48.870769 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.870756    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6d509\"/\"default-dockercfg-kh6mz\""
Sep 18 19:05:48.870823 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.870752    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-a6d509\"/\"rate-test-model-a6d509-kserve-self-signed-certs\""
Sep 18 19:05:48.878700 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.878679    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"]
Sep 18 19:05:48.921290 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.921253    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f36834eb-5287-4042-a850-52f9992ada74-tls-certs\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:48.921441 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.921319    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-home\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:48.921441 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.921365    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgdzd\" (UniqueName: \"kubernetes.io/projected/f36834eb-5287-4042-a850-52f9992ada74-kube-api-access-lgdzd\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:48.921441 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.921420    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-model-cache\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:48.921572 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.921470    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-tmp-dir\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:48.921572 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:48.921495    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-dshm\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:49.022349 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.022315    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-home\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:49.022349 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.022353    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgdzd\" (UniqueName: \"kubernetes.io/projected/f36834eb-5287-4042-a850-52f9992ada74-kube-api-access-lgdzd\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:49.022568 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.022474    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-model-cache\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:49.022613 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.022565    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-tmp-dir\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:49.022613 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.022592    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-dshm\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:49.022690 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.022672    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f36834eb-5287-4042-a850-52f9992ada74-tls-certs\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:49.022736 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.022702    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-home\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:49.022814 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.022794    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-model-cache\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:49.022929 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.022909    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-tmp-dir\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:49.024923 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.024903    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-dshm\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:49.025160 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.025143    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f36834eb-5287-4042-a850-52f9992ada74-tls-certs\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:49.032227 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.032178    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgdzd\" (UniqueName: \"kubernetes.io/projected/f36834eb-5287-4042-a850-52f9992ada74-kube-api-access-lgdzd\") pod \"rate-test-model-a6d509-kserve-7549d47759-f2frk\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") " pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:49.175539 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.175455    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:49.451987 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.451954    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"]
Sep 18 19:05:49.453178 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:05:49.453154    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf36834eb_5287_4042_a850_52f9992ada74.slice/crio-d3e7121bea173f929b36456e41b7ddf945c261be414c8acbacf53a6edb3680ce WatchSource:0}: Error finding container d3e7121bea173f929b36456e41b7ddf945c261be414c8acbacf53a6edb3680ce: Status 404 returned error can't find the container with id d3e7121bea173f929b36456e41b7ddf945c261be414c8acbacf53a6edb3680ce
Sep 18 19:05:49.565861 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.565826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk" event={"ID":"f36834eb-5287-4042-a850-52f9992ada74","Type":"ContainerStarted","Data":"dcfbdeedf8f3d0d5c486f017796814c58d9bd9afc4cc805c4aeb443120026f7f"}
Sep 18 19:05:49.566027 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.565871    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk" event={"ID":"f36834eb-5287-4042-a850-52f9992ada74","Type":"ContainerStarted","Data":"d3e7121bea173f929b36456e41b7ddf945c261be414c8acbacf53a6edb3680ce"}
Sep 18 19:05:49.583616 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:49.583557    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk" podStartSLOduration=1.583534912 podStartE2EDuration="1.583534912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:05:49.582162449 +0000 UTC m=+1609.018741291" watchObservedRunningTime="2026-09-18 19:05:49.583534912 +0000 UTC m=+1609.020113756"
Sep 18 19:05:50.575881 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:50.575851    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c8f5988-ad2e-472d-bfec-32425b822948" path="/var/lib/kubelet/pods/1c8f5988-ad2e-472d-bfec-32425b822948/volumes"
Sep 18 19:05:53.402733 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:53.402700    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-880bee-29829300-d665s"]
Sep 18 19:05:53.405289 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:53.405265    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-880bee-29829300-d665s"]
Sep 18 19:05:53.630760 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:53.630721    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24"]
Sep 18 19:05:53.633234 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:53.633214    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-880bee-zxr24"]
Sep 18 19:05:54.575370 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:54.575336    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33fadacc-d28f-4cb1-976a-c908648db2c8" path="/var/lib/kubelet/pods/33fadacc-d28f-4cb1-976a-c908648db2c8/volumes"
Sep 18 19:05:54.575749 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:54.575641    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="444696b0-826c-4fe2-b096-52ed6c0ade7d" path="/var/lib/kubelet/pods/444696b0-826c-4fe2-b096-52ed6c0ade7d/volumes"
Sep 18 19:05:59.176317 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:59.176275    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:59.176706 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:59.176333    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:59.189146 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:59.189117    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:05:59.648730 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:05:59.648694    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:06:00.308260 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:00.308223    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5"]
Sep 18 19:06:00.311992 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:00.311970    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5"
Sep 18 19:06:00.314316 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:00.314297    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:06:00.314410 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:00.314334    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 19:06:00.320791 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:00.320602    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5"]
Sep 18 19:06:00.433509 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:00.433477    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8137d3af-36fd-4bb6-8b26-7b9a53b93421-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5\" (UID: \"8137d3af-36fd-4bb6-8b26-7b9a53b93421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5"
Sep 18 19:06:00.534671 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:00.534638    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8137d3af-36fd-4bb6-8b26-7b9a53b93421-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5\" (UID: \"8137d3af-36fd-4bb6-8b26-7b9a53b93421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5"
Sep 18 19:06:00.535332 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:00.535311    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8137d3af-36fd-4bb6-8b26-7b9a53b93421-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5\" (UID: \"8137d3af-36fd-4bb6-8b26-7b9a53b93421\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5"
Sep 18 19:06:00.624108 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:00.624074    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5"
Sep 18 19:06:00.897113 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:00.897088    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5"]
Sep 18 19:06:00.898782 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:06:00.898741    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8137d3af_36fd_4bb6_8b26_7b9a53b93421.slice/crio-caf2cefcc45811ee1497f00d2db9b98fab389ff3b5d902cd09d5552a117f0ed5 WatchSource:0}: Error finding container caf2cefcc45811ee1497f00d2db9b98fab389ff3b5d902cd09d5552a117f0ed5: Status 404 returned error can't find the container with id caf2cefcc45811ee1497f00d2db9b98fab389ff3b5d902cd09d5552a117f0ed5
Sep 18 19:06:01.653108 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:01.653066    2558 generic.go:356] "Generic (PLEG): container finished" podID="8137d3af-36fd-4bb6-8b26-7b9a53b93421" containerID="d361ca7feefc6aaf30aced2a7bc593588de7dea740380355f925851d6ef344d2" exitCode=0
Sep 18 19:06:01.653528 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:01.653148    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5" event={"ID":"8137d3af-36fd-4bb6-8b26-7b9a53b93421","Type":"ContainerDied","Data":"d361ca7feefc6aaf30aced2a7bc593588de7dea740380355f925851d6ef344d2"}
Sep 18 19:06:01.653528 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:01.653223    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5" event={"ID":"8137d3af-36fd-4bb6-8b26-7b9a53b93421","Type":"ContainerStarted","Data":"caf2cefcc45811ee1497f00d2db9b98fab389ff3b5d902cd09d5552a117f0ed5"}
Sep 18 19:06:02.797528 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:02.797496    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5"
Sep 18 19:06:02.857760 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:02.857729    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8137d3af-36fd-4bb6-8b26-7b9a53b93421-maas-api-service-ca\") pod \"8137d3af-36fd-4bb6-8b26-7b9a53b93421\" (UID: \"8137d3af-36fd-4bb6-8b26-7b9a53b93421\") "
Sep 18 19:06:02.858133 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:02.858113    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8137d3af-36fd-4bb6-8b26-7b9a53b93421-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8137d3af-36fd-4bb6-8b26-7b9a53b93421" (UID: "8137d3af-36fd-4bb6-8b26-7b9a53b93421"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:06:02.958750 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:02.958674    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8137d3af-36fd-4bb6-8b26-7b9a53b93421-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:03.007940 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:03.007905    2558 scope.go:117] "RemoveContainer" containerID="d859e1a03750e51e43d41e65d5536557c806d8b94ab7bacd7490723e07d7d779"
Sep 18 19:06:03.017429 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:03.017402    2558 scope.go:117] "RemoveContainer" containerID="07a56c7583d10d51877bec2ab21ba1a5618c16a530324a1a359a2c231fafabc7"
Sep 18 19:06:03.025999 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:03.025977    2558 scope.go:117] "RemoveContainer" containerID="4480af971734d16e424db84df63ab751e35524c3d09997449c57d889ec32806f"
Sep 18 19:06:03.669790 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:03.669759    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5"
Sep 18 19:06:03.669983 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:03.669806    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5" event={"ID":"8137d3af-36fd-4bb6-8b26-7b9a53b93421","Type":"ContainerDied","Data":"caf2cefcc45811ee1497f00d2db9b98fab389ff3b5d902cd09d5552a117f0ed5"}
Sep 18 19:06:03.669983 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:03.669859    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caf2cefcc45811ee1497f00d2db9b98fab389ff3b5d902cd09d5552a117f0ed5"
Sep 18 19:06:11.010180 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:11.010142    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5"]
Sep 18 19:06:11.012591 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:11.012564    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-351045-w2sh5"]
Sep 18 19:06:12.575508 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:12.575478    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8137d3af-36fd-4bb6-8b26-7b9a53b93421" path="/var/lib/kubelet/pods/8137d3af-36fd-4bb6-8b26-7b9a53b93421/volumes"
Sep 18 19:06:17.475342 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:17.475305    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"]
Sep 18 19:06:17.475894 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:17.475557    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4" podUID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerName="istio-proxy" containerID="cri-o://4673b9343bbd3626c0c16417275c4ae77c6162a2e33080ef0353c82a5a57e90f" gracePeriod=30
Sep 18 19:06:18.748474 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:18.748437    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 19:06:18.748952 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:18.748496    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4" podUID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:06:22.726162 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.726136    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 19:06:22.793540 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.793453    2558 generic.go:356] "Generic (PLEG): container finished" podID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerID="4673b9343bbd3626c0c16417275c4ae77c6162a2e33080ef0353c82a5a57e90f" exitCode=0
Sep 18 19:06:22.793540 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.793502    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"
Sep 18 19:06:22.793778 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.793547    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4" event={"ID":"ef653752-c2ed-41cc-89ed-ee218cdb1d8a","Type":"ContainerDied","Data":"4673b9343bbd3626c0c16417275c4ae77c6162a2e33080ef0353c82a5a57e90f"}
Sep 18 19:06:22.793778 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.793608    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4" event={"ID":"ef653752-c2ed-41cc-89ed-ee218cdb1d8a","Type":"ContainerDied","Data":"03ad71c448120740727c86dc6faa1278232cdb9b9668600c70450ea54b83c705"}
Sep 18 19:06:22.793778 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.793638    2558 scope.go:117] "RemoveContainer" containerID="4673b9343bbd3626c0c16417275c4ae77c6162a2e33080ef0353c82a5a57e90f"
Sep 18 19:06:22.802975 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.802959    2558 scope.go:117] "RemoveContainer" containerID="381a3598e5cbe4238939274e6fb5eacc70fa94591877cfb112d01dccb6b54205"
Sep 18 19:06:22.811713 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.811697    2558 scope.go:117] "RemoveContainer" containerID="4673b9343bbd3626c0c16417275c4ae77c6162a2e33080ef0353c82a5a57e90f"
Sep 18 19:06:22.811986 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:06:22.811966    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4673b9343bbd3626c0c16417275c4ae77c6162a2e33080ef0353c82a5a57e90f\": container with ID starting with 4673b9343bbd3626c0c16417275c4ae77c6162a2e33080ef0353c82a5a57e90f not found: ID does not exist" containerID="4673b9343bbd3626c0c16417275c4ae77c6162a2e33080ef0353c82a5a57e90f"
Sep 18 19:06:22.812065 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.812000    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4673b9343bbd3626c0c16417275c4ae77c6162a2e33080ef0353c82a5a57e90f"} err="failed to get container status \"4673b9343bbd3626c0c16417275c4ae77c6162a2e33080ef0353c82a5a57e90f\": rpc error: code = NotFound desc = could not find container \"4673b9343bbd3626c0c16417275c4ae77c6162a2e33080ef0353c82a5a57e90f\": container with ID starting with 4673b9343bbd3626c0c16417275c4ae77c6162a2e33080ef0353c82a5a57e90f not found: ID does not exist"
Sep 18 19:06:22.812065 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.812026    2558 scope.go:117] "RemoveContainer" containerID="381a3598e5cbe4238939274e6fb5eacc70fa94591877cfb112d01dccb6b54205"
Sep 18 19:06:22.812315 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:06:22.812297    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"381a3598e5cbe4238939274e6fb5eacc70fa94591877cfb112d01dccb6b54205\": container with ID starting with 381a3598e5cbe4238939274e6fb5eacc70fa94591877cfb112d01dccb6b54205 not found: ID does not exist" containerID="381a3598e5cbe4238939274e6fb5eacc70fa94591877cfb112d01dccb6b54205"
Sep 18 19:06:22.812362 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.812322    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"381a3598e5cbe4238939274e6fb5eacc70fa94591877cfb112d01dccb6b54205"} err="failed to get container status \"381a3598e5cbe4238939274e6fb5eacc70fa94591877cfb112d01dccb6b54205\": rpc error: code = NotFound desc = could not find container \"381a3598e5cbe4238939274e6fb5eacc70fa94591877cfb112d01dccb6b54205\": container with ID starting with 381a3598e5cbe4238939274e6fb5eacc70fa94591877cfb112d01dccb6b54205 not found: ID does not exist"
Sep 18 19:06:22.859686 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.859639    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istiod-ca-cert\") pod \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") "
Sep 18 19:06:22.859861 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.859705    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-token\") pod \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") "
Sep 18 19:06:22.859861 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.859750    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx66k\" (UniqueName: \"kubernetes.io/projected/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-kube-api-access-jx66k\") pod \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") "
Sep 18 19:06:22.859861 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.859780    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-envoy\") pod \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") "
Sep 18 19:06:22.859861 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.859802    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-data\") pod \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") "
Sep 18 19:06:22.860094 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.859868    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-podinfo\") pod \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") "
Sep 18 19:06:22.860094 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.859908    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-credential-socket\") pod \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") "
Sep 18 19:06:22.860094 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.859941    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-workload-socket\") pod \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") "
Sep 18 19:06:22.860094 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.860028    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-workload-certs\") pod \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\" (UID: \"ef653752-c2ed-41cc-89ed-ee218cdb1d8a\") "
Sep 18 19:06:22.860335 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.860252    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ef653752-c2ed-41cc-89ed-ee218cdb1d8a" (UID: "ef653752-c2ed-41cc-89ed-ee218cdb1d8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:06:22.860335 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.860272    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ef653752-c2ed-41cc-89ed-ee218cdb1d8a" (UID: "ef653752-c2ed-41cc-89ed-ee218cdb1d8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:06:22.860482 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.860451    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-data" (OuterVolumeSpecName: "istio-data") pod "ef653752-c2ed-41cc-89ed-ee218cdb1d8a" (UID: "ef653752-c2ed-41cc-89ed-ee218cdb1d8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:06:22.860549 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.860475    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ef653752-c2ed-41cc-89ed-ee218cdb1d8a" (UID: "ef653752-c2ed-41cc-89ed-ee218cdb1d8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:06:22.860549 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.860466    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ef653752-c2ed-41cc-89ed-ee218cdb1d8a" (UID: "ef653752-c2ed-41cc-89ed-ee218cdb1d8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:06:22.860549 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.860462    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:22.860549 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.860516    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:22.862039 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.862015    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-kube-api-access-jx66k" (OuterVolumeSpecName: "kube-api-access-jx66k") pod "ef653752-c2ed-41cc-89ed-ee218cdb1d8a" (UID: "ef653752-c2ed-41cc-89ed-ee218cdb1d8a"). InnerVolumeSpecName "kube-api-access-jx66k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:06:22.862136 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.862100    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ef653752-c2ed-41cc-89ed-ee218cdb1d8a" (UID: "ef653752-c2ed-41cc-89ed-ee218cdb1d8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:06:22.862429 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.862407    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-token" (OuterVolumeSpecName: "istio-token") pod "ef653752-c2ed-41cc-89ed-ee218cdb1d8a" (UID: "ef653752-c2ed-41cc-89ed-ee218cdb1d8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:06:22.862497 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.862426    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ef653752-c2ed-41cc-89ed-ee218cdb1d8a" (UID: "ef653752-c2ed-41cc-89ed-ee218cdb1d8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 19:06:22.961242 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.961211    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:22.961242 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.961237    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:22.961242 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.961246    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx66k\" (UniqueName: \"kubernetes.io/projected/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-kube-api-access-jx66k\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:22.961484 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.961255    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:22.961484 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.961263    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:22.961484 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.961270    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:22.961484 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:22.961279    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef653752-c2ed-41cc-89ed-ee218cdb1d8a-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:23.119590 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:23.119560    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"]
Sep 18 19:06:23.125845 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:23.125821    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-351045-openshift-default-54fcf577c4-8t7g4"]
Sep 18 19:06:24.575871 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:24.575837    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" path="/var/lib/kubelet/pods/ef653752-c2ed-41cc-89ed-ee218cdb1d8a/volumes"
Sep 18 19:06:27.951818 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:27.951773    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"]
Sep 18 19:06:28.896705 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:28.896674    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"]
Sep 18 19:06:29.655204 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.655153    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"]
Sep 18 19:06:29.655756 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.655739    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerName="istio-proxy"
Sep 18 19:06:29.655821 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.655759    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerName="istio-proxy"
Sep 18 19:06:29.655821 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.655780    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerName="istio-proxy"
Sep 18 19:06:29.655821 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.655790    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerName="istio-proxy"
Sep 18 19:06:29.655821 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.655818    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8137d3af-36fd-4bb6-8b26-7b9a53b93421" containerName="revoke-keys"
Sep 18 19:06:29.656030 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.655827    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8137d3af-36fd-4bb6-8b26-7b9a53b93421" containerName="revoke-keys"
Sep 18 19:06:29.656030 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.655908    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerName="istio-proxy"
Sep 18 19:06:29.656030 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.655918    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8137d3af-36fd-4bb6-8b26-7b9a53b93421" containerName="revoke-keys"
Sep 18 19:06:29.656030 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.655929    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef653752-c2ed-41cc-89ed-ee218cdb1d8a" containerName="istio-proxy"
Sep 18 19:06:29.659333 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.659302    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.663067 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.663042    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0991d8-openshift-default-dockercfg-zqb8t\""
Sep 18 19:06:29.673133 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.673104    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"]
Sep 18 19:06:29.717473 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.717435    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-podinfo\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.717650 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.717480    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-envoy\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.717650 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.717530    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-workload-socket\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.717650 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.717560    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istiod-ca-cert\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.717650 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.717613    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-workload-certs\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.717874 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.717736    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-credential-socket\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.717874 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.717774    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcsgr\" (UniqueName: \"kubernetes.io/projected/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-kube-api-access-pcsgr\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.717874 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.717818    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-data\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.717874 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.717864    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-token\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.818991 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.818954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-podinfo\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.818991 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.819001    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-envoy\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.819285 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.819132    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-workload-socket\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.819285 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.819198    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istiod-ca-cert\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.819285 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.819269    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-workload-certs\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.819442 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.819352    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-credential-socket\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.819442 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.819380    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcsgr\" (UniqueName: \"kubernetes.io/projected/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-kube-api-access-pcsgr\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.819442 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.819413    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-data\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.819594 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.819470    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-token\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.819594 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.819514    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-workload-socket\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.819692 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.819610    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-workload-certs\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.819875 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.819852    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-data\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.819948 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.819931    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istiod-ca-cert\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.819987 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.819929    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-credential-socket\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.821683 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.821659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-envoy\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.821791 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.821719    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-podinfo\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.830930 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.830902    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-token\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.831097 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.831082    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcsgr\" (UniqueName: \"kubernetes.io/projected/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-kube-api-access-pcsgr\") pod \"e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") " pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:29.969417 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:29.969324    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:30.269111 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:30.269080    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"]
Sep 18 19:06:30.270478 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:06:30.270450    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4e8663_2d6b_4494_90d2_34a56f47c4aa.slice/crio-e3c1b98a75a08668df8d24d3c653e5f40a1bfe5b80827fe8fa7a454477d009f8 WatchSource:0}: Error finding container e3c1b98a75a08668df8d24d3c653e5f40a1bfe5b80827fe8fa7a454477d009f8: Status 404 returned error can't find the container with id e3c1b98a75a08668df8d24d3c653e5f40a1bfe5b80827fe8fa7a454477d009f8
Sep 18 19:06:30.272803 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:30.272773    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:06:30.272866 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:30.272836    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:06:30.272902 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:30.272866    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:06:30.845986 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:30.845947    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2" event={"ID":"4e4e8663-2d6b-4494-90d2-34a56f47c4aa","Type":"ContainerStarted","Data":"0eb1a990905ed76eb28d3f003ab81acd11ccec57567e05717e3a6c9635101409"}
Sep 18 19:06:30.846393 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:30.846046    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2" event={"ID":"4e4e8663-2d6b-4494-90d2-34a56f47c4aa","Type":"ContainerStarted","Data":"e3c1b98a75a08668df8d24d3c653e5f40a1bfe5b80827fe8fa7a454477d009f8"}
Sep 18 19:06:30.866036 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:30.865981    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2" podStartSLOduration=1.865963395 podStartE2EDuration="1.865963395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:06:30.865735614 +0000 UTC m=+1650.302314457" watchObservedRunningTime="2026-09-18 19:06:30.865963395 +0000 UTC m=+1650.302542237"
Sep 18 19:06:30.969770 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:30.969736    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:31.975221 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:31.975175    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:32.858643 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:32.858614    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:32.859778 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:32.859755    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:42.971809 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:42.971754    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7" podUID="71bdd860-9a46-4cac-ad86-fa7dec06f902" containerName="main" containerID="cri-o://62e332cc380698c312d723a21e75fd6291e77e40a1194ff00f4a407eedf5a70a" gracePeriod=45
Sep 18 19:06:43.270539 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.270199    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:06:43.369567 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.369528    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvprx\" (UniqueName: \"kubernetes.io/projected/71bdd860-9a46-4cac-ad86-fa7dec06f902-kube-api-access-zvprx\") pod \"71bdd860-9a46-4cac-ad86-fa7dec06f902\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") "
Sep 18 19:06:43.369816 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.369796    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-home\") pod \"71bdd860-9a46-4cac-ad86-fa7dec06f902\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") "
Sep 18 19:06:43.369976 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.369959    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-dshm\") pod \"71bdd860-9a46-4cac-ad86-fa7dec06f902\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") "
Sep 18 19:06:43.370125 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.370107    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71bdd860-9a46-4cac-ad86-fa7dec06f902-tls-certs\") pod \"71bdd860-9a46-4cac-ad86-fa7dec06f902\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") "
Sep 18 19:06:43.370300 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.370285    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-tmp-dir\") pod \"71bdd860-9a46-4cac-ad86-fa7dec06f902\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") "
Sep 18 19:06:43.370555 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.370537    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-model-cache\") pod \"71bdd860-9a46-4cac-ad86-fa7dec06f902\" (UID: \"71bdd860-9a46-4cac-ad86-fa7dec06f902\") "
Sep 18 19:06:43.371251 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.371229    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-model-cache" (OuterVolumeSpecName: "model-cache") pod "71bdd860-9a46-4cac-ad86-fa7dec06f902" (UID: "71bdd860-9a46-4cac-ad86-fa7dec06f902"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:06:43.373011 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.372749    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71bdd860-9a46-4cac-ad86-fa7dec06f902" (UID: "71bdd860-9a46-4cac-ad86-fa7dec06f902"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:06:43.373011 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.372949    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-home" (OuterVolumeSpecName: "home") pod "71bdd860-9a46-4cac-ad86-fa7dec06f902" (UID: "71bdd860-9a46-4cac-ad86-fa7dec06f902"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:06:43.373860 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.373833    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71bdd860-9a46-4cac-ad86-fa7dec06f902-kube-api-access-zvprx" (OuterVolumeSpecName: "kube-api-access-zvprx") pod "71bdd860-9a46-4cac-ad86-fa7dec06f902" (UID: "71bdd860-9a46-4cac-ad86-fa7dec06f902"). InnerVolumeSpecName "kube-api-access-zvprx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:06:43.374596 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.374568    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-dshm" (OuterVolumeSpecName: "dshm") pod "71bdd860-9a46-4cac-ad86-fa7dec06f902" (UID: "71bdd860-9a46-4cac-ad86-fa7dec06f902"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:06:43.374801 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.374781    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71bdd860-9a46-4cac-ad86-fa7dec06f902-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71bdd860-9a46-4cac-ad86-fa7dec06f902" (UID: "71bdd860-9a46-4cac-ad86-fa7dec06f902"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:06:43.472687 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.472639    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-model-cache\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:43.472687 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.472684    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvprx\" (UniqueName: \"kubernetes.io/projected/71bdd860-9a46-4cac-ad86-fa7dec06f902-kube-api-access-zvprx\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:43.472967 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.472701    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-home\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:43.472967 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.472715    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-dshm\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:43.472967 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.472728    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71bdd860-9a46-4cac-ad86-fa7dec06f902-tls-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:43.472967 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.472744    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71bdd860-9a46-4cac-ad86-fa7dec06f902-tmp-dir\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:43.915725 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.915672    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk" podUID="f36834eb-5287-4042-a850-52f9992ada74" containerName="main" containerID="cri-o://dcfbdeedf8f3d0d5c486f017796814c58d9bd9afc4cc805c4aeb443120026f7f" gracePeriod=45
Sep 18 19:06:43.935095 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.935068    2558 generic.go:356] "Generic (PLEG): container finished" podID="71bdd860-9a46-4cac-ad86-fa7dec06f902" containerID="62e332cc380698c312d723a21e75fd6291e77e40a1194ff00f4a407eedf5a70a" exitCode=0
Sep 18 19:06:43.935222 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.935120    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"
Sep 18 19:06:43.935222 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.935144    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7" event={"ID":"71bdd860-9a46-4cac-ad86-fa7dec06f902","Type":"ContainerDied","Data":"62e332cc380698c312d723a21e75fd6291e77e40a1194ff00f4a407eedf5a70a"}
Sep 18 19:06:43.935222 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.935173    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7" event={"ID":"71bdd860-9a46-4cac-ad86-fa7dec06f902","Type":"ContainerDied","Data":"766c2a263d84484607373bf3e7b7e1b28dd5fefee66398549806a261d4453852"}
Sep 18 19:06:43.935222 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.935205    2558 scope.go:117] "RemoveContainer" containerID="62e332cc380698c312d723a21e75fd6291e77e40a1194ff00f4a407eedf5a70a"
Sep 18 19:06:43.981642 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.981604    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"]
Sep 18 19:06:43.985383 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:43.985356    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-298125/rate-test-model-298125-kserve-66557f479-fsrg7"]
Sep 18 19:06:44.027123 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.026967    2558 scope.go:117] "RemoveContainer" containerID="62e332cc380698c312d723a21e75fd6291e77e40a1194ff00f4a407eedf5a70a"
Sep 18 19:06:44.027367 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:06:44.027346    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e332cc380698c312d723a21e75fd6291e77e40a1194ff00f4a407eedf5a70a\": container with ID starting with 62e332cc380698c312d723a21e75fd6291e77e40a1194ff00f4a407eedf5a70a not found: ID does not exist" containerID="62e332cc380698c312d723a21e75fd6291e77e40a1194ff00f4a407eedf5a70a"
Sep 18 19:06:44.027442 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.027382    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e332cc380698c312d723a21e75fd6291e77e40a1194ff00f4a407eedf5a70a"} err="failed to get container status \"62e332cc380698c312d723a21e75fd6291e77e40a1194ff00f4a407eedf5a70a\": rpc error: code = NotFound desc = could not find container \"62e332cc380698c312d723a21e75fd6291e77e40a1194ff00f4a407eedf5a70a\": container with ID starting with 62e332cc380698c312d723a21e75fd6291e77e40a1194ff00f4a407eedf5a70a not found: ID does not exist"
Sep 18 19:06:44.203095 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.203070    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:06:44.280805 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.280775    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-dshm\") pod \"f36834eb-5287-4042-a850-52f9992ada74\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") "
Sep 18 19:06:44.281009 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.280819    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f36834eb-5287-4042-a850-52f9992ada74-tls-certs\") pod \"f36834eb-5287-4042-a850-52f9992ada74\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") "
Sep 18 19:06:44.281009 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.280868    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-model-cache\") pod \"f36834eb-5287-4042-a850-52f9992ada74\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") "
Sep 18 19:06:44.281009 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.280973    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgdzd\" (UniqueName: \"kubernetes.io/projected/f36834eb-5287-4042-a850-52f9992ada74-kube-api-access-lgdzd\") pod \"f36834eb-5287-4042-a850-52f9992ada74\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") "
Sep 18 19:06:44.281206 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.281031    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-tmp-dir\") pod \"f36834eb-5287-4042-a850-52f9992ada74\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") "
Sep 18 19:06:44.281206 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.281057    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-home\") pod \"f36834eb-5287-4042-a850-52f9992ada74\" (UID: \"f36834eb-5287-4042-a850-52f9992ada74\") "
Sep 18 19:06:44.281206 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.281088    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-model-cache" (OuterVolumeSpecName: "model-cache") pod "f36834eb-5287-4042-a850-52f9992ada74" (UID: "f36834eb-5287-4042-a850-52f9992ada74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:06:44.281381 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.281272    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-home" (OuterVolumeSpecName: "home") pod "f36834eb-5287-4042-a850-52f9992ada74" (UID: "f36834eb-5287-4042-a850-52f9992ada74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:06:44.281381 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.281287    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f36834eb-5287-4042-a850-52f9992ada74" (UID: "f36834eb-5287-4042-a850-52f9992ada74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:06:44.281492 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.281463    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-tmp-dir\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:44.281550 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.281492    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-home\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:44.281550 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.281500    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-model-cache\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:44.283049 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.283022    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36834eb-5287-4042-a850-52f9992ada74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f36834eb-5287-4042-a850-52f9992ada74" (UID: "f36834eb-5287-4042-a850-52f9992ada74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:06:44.283049 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.283036    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-dshm" (OuterVolumeSpecName: "dshm") pod "f36834eb-5287-4042-a850-52f9992ada74" (UID: "f36834eb-5287-4042-a850-52f9992ada74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:06:44.283243 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.283063    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36834eb-5287-4042-a850-52f9992ada74-kube-api-access-lgdzd" (OuterVolumeSpecName: "kube-api-access-lgdzd") pod "f36834eb-5287-4042-a850-52f9992ada74" (UID: "f36834eb-5287-4042-a850-52f9992ada74"). InnerVolumeSpecName "kube-api-access-lgdzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:06:44.382079 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.382041    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f36834eb-5287-4042-a850-52f9992ada74-tls-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:44.382079 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.382077    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgdzd\" (UniqueName: \"kubernetes.io/projected/f36834eb-5287-4042-a850-52f9992ada74-kube-api-access-lgdzd\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:44.382079 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.382088    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f36834eb-5287-4042-a850-52f9992ada74-dshm\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:06:44.578797 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.578721    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71bdd860-9a46-4cac-ad86-fa7dec06f902" path="/var/lib/kubelet/pods/71bdd860-9a46-4cac-ad86-fa7dec06f902/volumes"
Sep 18 19:06:44.943728 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.943696    2558 generic.go:356] "Generic (PLEG): container finished" podID="f36834eb-5287-4042-a850-52f9992ada74" containerID="dcfbdeedf8f3d0d5c486f017796814c58d9bd9afc4cc805c4aeb443120026f7f" exitCode=0
Sep 18 19:06:44.943917 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.943743    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"
Sep 18 19:06:44.943917 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.943785    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk" event={"ID":"f36834eb-5287-4042-a850-52f9992ada74","Type":"ContainerDied","Data":"dcfbdeedf8f3d0d5c486f017796814c58d9bd9afc4cc805c4aeb443120026f7f"}
Sep 18 19:06:44.943917 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.943829    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk" event={"ID":"f36834eb-5287-4042-a850-52f9992ada74","Type":"ContainerDied","Data":"d3e7121bea173f929b36456e41b7ddf945c261be414c8acbacf53a6edb3680ce"}
Sep 18 19:06:44.943917 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.943848    2558 scope.go:117] "RemoveContainer" containerID="dcfbdeedf8f3d0d5c486f017796814c58d9bd9afc4cc805c4aeb443120026f7f"
Sep 18 19:06:44.962500 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.962460    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"]
Sep 18 19:06:44.962675 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.962641    2558 scope.go:117] "RemoveContainer" containerID="dcfbdeedf8f3d0d5c486f017796814c58d9bd9afc4cc805c4aeb443120026f7f"
Sep 18 19:06:44.966682 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:06:44.966648    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcfbdeedf8f3d0d5c486f017796814c58d9bd9afc4cc805c4aeb443120026f7f\": container with ID starting with dcfbdeedf8f3d0d5c486f017796814c58d9bd9afc4cc805c4aeb443120026f7f not found: ID does not exist" containerID="dcfbdeedf8f3d0d5c486f017796814c58d9bd9afc4cc805c4aeb443120026f7f"
Sep 18 19:06:44.966802 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.966690    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfbdeedf8f3d0d5c486f017796814c58d9bd9afc4cc805c4aeb443120026f7f"} err="failed to get container status \"dcfbdeedf8f3d0d5c486f017796814c58d9bd9afc4cc805c4aeb443120026f7f\": rpc error: code = NotFound desc = could not find container \"dcfbdeedf8f3d0d5c486f017796814c58d9bd9afc4cc805c4aeb443120026f7f\": container with ID starting with dcfbdeedf8f3d0d5c486f017796814c58d9bd9afc4cc805c4aeb443120026f7f not found: ID does not exist"
Sep 18 19:06:44.967421 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:44.967396    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a6d509/rate-test-model-a6d509-kserve-7549d47759-f2frk"]
Sep 18 19:06:46.578905 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:46.578867    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f36834eb-5287-4042-a850-52f9992ada74" path="/var/lib/kubelet/pods/f36834eb-5287-4042-a850-52f9992ada74/volumes"
Sep 18 19:06:48.899344 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:48.899297    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 18 19:06:48.899893 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:48.899386    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2" podUID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 19:06:50.992567 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:50.992535    2558 generic.go:356] "Generic (PLEG): container finished" podID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" containerID="0eb1a990905ed76eb28d3f003ab81acd11ccec57567e05717e3a6c9635101409" exitCode=0
Sep 18 19:06:50.992992 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:50.992612    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2" event={"ID":"4e4e8663-2d6b-4494-90d2-34a56f47c4aa","Type":"ContainerDied","Data":"0eb1a990905ed76eb28d3f003ab81acd11ccec57567e05717e3a6c9635101409"}
Sep 18 19:06:50.993036 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:50.992996    2558 scope.go:117] "RemoveContainer" containerID="0eb1a990905ed76eb28d3f003ab81acd11ccec57567e05717e3a6c9635101409"
Sep 18 19:06:50.994135 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:50.994105    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:06:50.994211 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:50.994180    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:06:50.994259 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:50.994243    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:06:52.000623 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:52.000584    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2" event={"ID":"4e4e8663-2d6b-4494-90d2-34a56f47c4aa","Type":"ContainerStarted","Data":"571afc33ae9905aabbcd8754b0ed4e9f6c83730795c2b04ffd890f9a5b7591cc"}
Sep 18 19:06:52.970066 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:52.970034    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:52.970370 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:52.970337    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Sep 18 19:06:52.970665 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:52.970635    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2" podUID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Sep 18 19:06:53.970227 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:53.970164    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Sep 18 19:06:53.970734 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:53.970252    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2" podUID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Sep 18 19:06:54.970395 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:54.970349    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Sep 18 19:06:54.971020 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:54.970972    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2" podUID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Sep 18 19:06:55.970410 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:55.970361    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Sep 18 19:06:55.970915 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:55.970452    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2" podUID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Sep 18 19:06:56.970374 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:56.970337    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Sep 18 19:06:56.970710 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:56.970665    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2" podUID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Sep 18 19:06:57.973355 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:57.973323    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:57.973846 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:57.973716    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:57.974516 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:57.974496    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:06:58.781704 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.781665    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"]
Sep 18 19:06:58.782314 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.782294    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36834eb-5287-4042-a850-52f9992ada74" containerName="main"
Sep 18 19:06:58.782368 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.782317    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36834eb-5287-4042-a850-52f9992ada74" containerName="main"
Sep 18 19:06:58.782368 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.782342    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71bdd860-9a46-4cac-ad86-fa7dec06f902" containerName="main"
Sep 18 19:06:58.782368 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.782351    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="71bdd860-9a46-4cac-ad86-fa7dec06f902" containerName="main"
Sep 18 19:06:58.782493 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.782444    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f36834eb-5287-4042-a850-52f9992ada74" containerName="main"
Sep 18 19:06:58.782493 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.782456    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="71bdd860-9a46-4cac-ad86-fa7dec06f902" containerName="main"
Sep 18 19:06:58.785602 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.785584    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:58.788077 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.788057    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d6246c-openshift-default-dockercfg-x9qdl\""
Sep 18 19:06:58.798174 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.798152    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"]
Sep 18 19:06:58.931102 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.931062    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvpmd\" (UniqueName: \"kubernetes.io/projected/c01ed0fa-e16d-4119-893b-24d93d5f70f5-kube-api-access-qvpmd\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:58.931306 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.931108    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-data\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:58.931306 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.931138    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-token\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:58.931306 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.931250    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istiod-ca-cert\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:58.931306 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.931292    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-podinfo\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:58.931454 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.931367    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-envoy\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:58.931454 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.931420    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-workload-certs\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:58.931517 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.931463    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-workload-socket\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:58.931517 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:58.931485    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-credential-socket\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.032349 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.032261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-workload-certs\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.032349 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.032315    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-workload-socket\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.032349 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.032337    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-credential-socket\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.032930 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.032385    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvpmd\" (UniqueName: \"kubernetes.io/projected/c01ed0fa-e16d-4119-893b-24d93d5f70f5-kube-api-access-qvpmd\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.032930 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.032406    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-data\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.032930 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.032430    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-token\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.032930 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.032465    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istiod-ca-cert\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.032930 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.032489    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-podinfo\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.032930 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.032710    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-envoy\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.032930 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.032790    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-data\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.032930 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.032720    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-workload-certs\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.033363 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.033057    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-workload-socket\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.033363 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.033141    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-credential-socket\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.033529 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.033500    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istiod-ca-cert\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.034914 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.034894    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-envoy\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.035260 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.035240    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-podinfo\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.042014 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.041986    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvpmd\" (UniqueName: \"kubernetes.io/projected/c01ed0fa-e16d-4119-893b-24d93d5f70f5-kube-api-access-qvpmd\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.042289 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.042270    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-token\") pod \"e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") " pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.097527 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.097494    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:06:59.381069 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.381043    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"]
Sep 18 19:06:59.381828 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:06:59.381804    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01ed0fa_e16d_4119_893b_24d93d5f70f5.slice/crio-c25ae67136f32365d93bb078df4cb6ed3e1d82cfb2b13309381aaf1104b66ed1 WatchSource:0}: Error finding container c25ae67136f32365d93bb078df4cb6ed3e1d82cfb2b13309381aaf1104b66ed1: Status 404 returned error can't find the container with id c25ae67136f32365d93bb078df4cb6ed3e1d82cfb2b13309381aaf1104b66ed1
Sep 18 19:06:59.384347 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.384307    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:06:59.384459 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.384389    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:06:59.384459 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:06:59.384434    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:07:00.057740 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:00.057704    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9" event={"ID":"c01ed0fa-e16d-4119-893b-24d93d5f70f5","Type":"ContainerStarted","Data":"b6af462059927b38a4b419d28fcff9dc833dc1cd31e177404bad7e28f68ae3fb"}
Sep 18 19:07:00.058119 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:00.057757    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9" event={"ID":"c01ed0fa-e16d-4119-893b-24d93d5f70f5","Type":"ContainerStarted","Data":"c25ae67136f32365d93bb078df4cb6ed3e1d82cfb2b13309381aaf1104b66ed1"}
Sep 18 19:07:00.086984 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:00.086932    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9" podStartSLOduration=2.086917072 podStartE2EDuration="2.086917072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:07:00.085168768 +0000 UTC m=+1679.521747611" watchObservedRunningTime="2026-09-18 19:07:00.086917072 +0000 UTC m=+1679.523495971"
Sep 18 19:07:00.097698 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:00.097672    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:07:00.102944 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:00.102921    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:07:01.065587 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:01.065559    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:07:01.066575 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:01.066548    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:07:03.078689 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:03.078660    2558 scope.go:117] "RemoveContainer" containerID="5feb46f7638f5ee1ebad2a5df58a697afb8439b9be48e43ac084d0a06dd9846d"
Sep 18 19:07:20.201594 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:20.201499    2558 generic.go:356] "Generic (PLEG): container finished" podID="c01ed0fa-e16d-4119-893b-24d93d5f70f5" containerID="b6af462059927b38a4b419d28fcff9dc833dc1cd31e177404bad7e28f68ae3fb" exitCode=0
Sep 18 19:07:20.201594 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:20.201570    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9" event={"ID":"c01ed0fa-e16d-4119-893b-24d93d5f70f5","Type":"ContainerDied","Data":"b6af462059927b38a4b419d28fcff9dc833dc1cd31e177404bad7e28f68ae3fb"}
Sep 18 19:07:20.202016 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:20.201987    2558 scope.go:117] "RemoveContainer" containerID="b6af462059927b38a4b419d28fcff9dc833dc1cd31e177404bad7e28f68ae3fb"
Sep 18 19:07:20.203102 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:20.203062    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:07:20.203168 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:20.203133    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:07:20.203221 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:20.203166    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:07:21.209590 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:21.209556    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9" event={"ID":"c01ed0fa-e16d-4119-893b-24d93d5f70f5","Type":"ContainerStarted","Data":"0258bc7637bf4e368eca082497e2ac6009e858ae9975f3fc9db66afe5a79a658"}
Sep 18 19:07:22.098562 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:22.098515    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:07:22.098838 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:22.098817    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Sep 18 19:07:22.098895 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:22.098865    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9" podUID="c01ed0fa-e16d-4119-893b-24d93d5f70f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Sep 18 19:07:23.098232 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:23.098171    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Sep 18 19:07:23.098648 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:23.098265    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9" podUID="c01ed0fa-e16d-4119-893b-24d93d5f70f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Sep 18 19:07:24.098448 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:24.098410    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Sep 18 19:07:24.098842 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:24.098472    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9" podUID="c01ed0fa-e16d-4119-893b-24d93d5f70f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Sep 18 19:07:25.098807 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:25.098769    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Sep 18 19:07:25.099304 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:25.098831    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9" podUID="c01ed0fa-e16d-4119-893b-24d93d5f70f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Sep 18 19:07:26.098156 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:26.098114    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Sep 18 19:07:26.098367 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:26.098176    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9" podUID="c01ed0fa-e16d-4119-893b-24d93d5f70f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Sep 18 19:07:27.102459 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:27.102426    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:07:27.102945 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:27.102801    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:07:27.103504 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:27.103482    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:07:37.131016 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:37.130987    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-bbxqv_ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1/payload-processing/0.log"
Sep 18 19:07:37.248099 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:07:37.248071    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-bbxqv_ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1/payload-processing/0.log"
Sep 18 19:08:00.857020 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:00.856984    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"]
Sep 18 19:08:00.860916 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:00.860898    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:00.863329 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:00.863303    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0991d8\"/\"openshift-service-ca.crt\""
Sep 18 19:08:00.863452 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:00.863364    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0991d8\"/\"kube-root-ca.crt\""
Sep 18 19:08:00.864198 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:00.864169    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0991d8\"/\"default-dockercfg-qcpd6\""
Sep 18 19:08:00.864258 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:00.864238    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0991d8\"/\"ipp-route-0991d8-kserve-self-signed-certs\""
Sep 18 19:08:00.870977 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:00.870956    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"]
Sep 18 19:08:00.958812 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:00.958773    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-dshm\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:00.958812 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:00.958810    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-home\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:00.959072 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:00.958883    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-tmp-dir\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:00.959072 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:00.958912    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcbrz\" (UniqueName: \"kubernetes.io/projected/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-kube-api-access-qcbrz\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:00.959072 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:00.958979    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-tls-certs\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:00.959072 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:00.959024    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-model-cache\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:01.060491 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:01.060447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-tmp-dir\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:01.061335 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:01.061285    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcbrz\" (UniqueName: \"kubernetes.io/projected/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-kube-api-access-qcbrz\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:01.062061 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:01.062039    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-tls-certs\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:01.062686 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:01.061131    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-tmp-dir\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:01.062842 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:01.062696    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-model-cache\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:01.062990 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:01.062973    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-model-cache\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:01.063245 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:01.063216    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-dshm\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:01.063350 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:01.063258    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-home\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:01.063653 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:01.063632    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-home\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:01.065816 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:01.065493    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-tls-certs\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:01.065816 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:01.065760    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-dshm\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:01.069975 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:01.069946    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcbrz\" (UniqueName: \"kubernetes.io/projected/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-kube-api-access-qcbrz\") pod \"ipp-route-0991d8-kserve-59c7fdc78c-rsxpd\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") " pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:01.171300 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:01.171180    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:01.448985 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:01.448953    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"]
Sep 18 19:08:01.449937 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:08:01.449904    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3a5b59c_bc56_4149_9eb8_bcc071e5e2ee.slice/crio-386df5e33555e53e3ad042ec557983fae9ff16b4a3ba786c7d2d1b16ffd56c09 WatchSource:0}: Error finding container 386df5e33555e53e3ad042ec557983fae9ff16b4a3ba786c7d2d1b16ffd56c09: Status 404 returned error can't find the container with id 386df5e33555e53e3ad042ec557983fae9ff16b4a3ba786c7d2d1b16ffd56c09
Sep 18 19:08:01.461001 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:01.460972    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd" event={"ID":"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee","Type":"ContainerStarted","Data":"386df5e33555e53e3ad042ec557983fae9ff16b4a3ba786c7d2d1b16ffd56c09"}
Sep 18 19:08:02.469842 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:02.469801    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd" event={"ID":"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee","Type":"ContainerStarted","Data":"2edf0fa09fb91f759cc7789cbd55c005644ce75179c04707e56017a06a1383d3"}
Sep 18 19:08:02.489006 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:02.488956    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd" podStartSLOduration=2.488941166 podStartE2EDuration="2.488941166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:08:02.487524465 +0000 UTC m=+1741.924103347" watchObservedRunningTime="2026-09-18 19:08:02.488941166 +0000 UTC m=+1741.925520007"
Sep 18 19:08:03.116804 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:03.116766    2558 scope.go:117] "RemoveContainer" containerID="3eaa7dba37fde18b7bc36867b89c00f97c6759d40c22e70281c679f1cf345120"
Sep 18 19:08:11.172028 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:11.171991    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:11.172496 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:11.172044    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:11.184665 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:11.184635    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:11.542381 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:11.542300    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:08:23.795653 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:23.795624    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-bbxqv_ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1/payload-processing/0.log"
Sep 18 19:08:23.917335 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:23.917311    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-bbxqv_ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1/payload-processing/0.log"
Sep 18 19:08:24.732678 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.732646    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"]
Sep 18 19:08:24.737779 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.737762    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.741139 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.741119    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-5855d4-openshift-default-dockercfg-f7fpl\""
Sep 18 19:08:24.753310 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.753287    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"]
Sep 18 19:08:24.808071 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.808010    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd206272-6e62-4898-9dce-6d53c37e6d6c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.808587 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.808129    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-credential-socket\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.808587 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.808225    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-envoy\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.808587 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.808256    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-workload-socket\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.808587 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.808302    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-workload-certs\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.808587 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.808332    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-podinfo\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.808587 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.808454    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-data\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.808587 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.808519    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4zr7\" (UniqueName: \"kubernetes.io/projected/fd206272-6e62-4898-9dce-6d53c37e6d6c-kube-api-access-s4zr7\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.808587 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.808564    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-token\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.909692 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.909653    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-token\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.909692 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.909697    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd206272-6e62-4898-9dce-6d53c37e6d6c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.909963 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.909734    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-credential-socket\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.909963 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.909770    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-envoy\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.909963 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.909788    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-workload-socket\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.909963 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.909818    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-workload-certs\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.909963 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.909835    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-podinfo\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.909963 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.909876    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-data\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.909963 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.909935    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4zr7\" (UniqueName: \"kubernetes.io/projected/fd206272-6e62-4898-9dce-6d53c37e6d6c-kube-api-access-s4zr7\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.910364 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.910341    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-workload-certs\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.910424 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.910360    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-workload-socket\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.910521 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.910493    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-credential-socket\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.910598 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.910525    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-data\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.910698 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.910677    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd206272-6e62-4898-9dce-6d53c37e6d6c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.912300 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.912277    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-envoy\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.912448 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.912432    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-podinfo\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.918920 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.918895    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-token\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:24.922054 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:24.922033    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4zr7\" (UniqueName: \"kubernetes.io/projected/fd206272-6e62-4898-9dce-6d53c37e6d6c-kube-api-access-s4zr7\") pod \"e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") " pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:25.051064 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:25.050972    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:25.333434 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:25.333409    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"]
Sep 18 19:08:25.335768 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:08:25.335743    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd206272_6e62_4898_9dce_6d53c37e6d6c.slice/crio-ba852a5bb63e174ca437628945702898802fc727a42a0f49687dbb43dd5abac9 WatchSource:0}: Error finding container ba852a5bb63e174ca437628945702898802fc727a42a0f49687dbb43dd5abac9: Status 404 returned error can't find the container with id ba852a5bb63e174ca437628945702898802fc727a42a0f49687dbb43dd5abac9
Sep 18 19:08:25.338124 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:25.338086    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:08:25.338231 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:25.338159    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:08:25.338281 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:25.338226    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:08:25.620529 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:25.620494    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk" event={"ID":"fd206272-6e62-4898-9dce-6d53c37e6d6c","Type":"ContainerStarted","Data":"5af8f5f104920d9aa3c4ba5f4921d2f660e2fe1052f91798cecc8340ae5f9afc"}
Sep 18 19:08:25.620713 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:25.620552    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk" event={"ID":"fd206272-6e62-4898-9dce-6d53c37e6d6c","Type":"ContainerStarted","Data":"ba852a5bb63e174ca437628945702898802fc727a42a0f49687dbb43dd5abac9"}
Sep 18 19:08:25.641614 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:25.641557    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk" podStartSLOduration=1.641541019 podStartE2EDuration="1.641541019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:08:25.639858085 +0000 UTC m=+1765.076436967" watchObservedRunningTime="2026-09-18 19:08:25.641541019 +0000 UTC m=+1765.078119861"
Sep 18 19:08:26.051446 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:26.051340    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:26.057411 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:26.057389    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:26.627350 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:26.627315    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:26.628462 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:26.628438    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:48.776970 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:48.776938    2558 generic.go:356] "Generic (PLEG): container finished" podID="fd206272-6e62-4898-9dce-6d53c37e6d6c" containerID="5af8f5f104920d9aa3c4ba5f4921d2f660e2fe1052f91798cecc8340ae5f9afc" exitCode=0
Sep 18 19:08:48.777437 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:48.777016    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk" event={"ID":"fd206272-6e62-4898-9dce-6d53c37e6d6c","Type":"ContainerDied","Data":"5af8f5f104920d9aa3c4ba5f4921d2f660e2fe1052f91798cecc8340ae5f9afc"}
Sep 18 19:08:48.777556 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:48.777539    2558 scope.go:117] "RemoveContainer" containerID="5af8f5f104920d9aa3c4ba5f4921d2f660e2fe1052f91798cecc8340ae5f9afc"
Sep 18 19:08:48.778748 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:48.778711    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:08:48.778871 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:48.778802    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:08:48.778871 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:48.778854    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 19:08:49.786124 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:49.786078    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk" event={"ID":"fd206272-6e62-4898-9dce-6d53c37e6d6c","Type":"ContainerStarted","Data":"ada5e7c06ad6ad150681edd9d65d70f19f4b15913c9833ba5095e424e599a2e7"}
Sep 18 19:08:50.051696 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:50.051598    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:50.051696 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:50.051656    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Sep 18 19:08:50.051942 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:50.051737    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk" podUID="fd206272-6e62-4898-9dce-6d53c37e6d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Sep 18 19:08:51.052200 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:51.051726    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Sep 18 19:08:51.052200 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:51.051797    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk" podUID="fd206272-6e62-4898-9dce-6d53c37e6d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Sep 18 19:08:52.052067 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:52.052028    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Sep 18 19:08:52.052414 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:52.052381    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk" podUID="fd206272-6e62-4898-9dce-6d53c37e6d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Sep 18 19:08:53.052328 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:53.052284    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Sep 18 19:08:53.052559 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:53.052370    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk" podUID="fd206272-6e62-4898-9dce-6d53c37e6d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Sep 18 19:08:53.211040 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:53.211006    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc"]
Sep 18 19:08:53.214738 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:53.214717    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc"
Sep 18 19:08:53.217293 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:53.217256    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:08:53.217401 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:53.217378    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 19:08:53.224098 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:53.224076    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc"]
Sep 18 19:08:53.282294 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:53.282252    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/104675b1-4c80-433d-b1a4-f28afa88700d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc\" (UID: \"104675b1-4c80-433d-b1a4-f28afa88700d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc"
Sep 18 19:08:53.383529 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:53.383489    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/104675b1-4c80-433d-b1a4-f28afa88700d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc\" (UID: \"104675b1-4c80-433d-b1a4-f28afa88700d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc"
Sep 18 19:08:53.384127 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:53.384108    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/104675b1-4c80-433d-b1a4-f28afa88700d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc\" (UID: \"104675b1-4c80-433d-b1a4-f28afa88700d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc"
Sep 18 19:08:53.526174 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:53.526141    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc"
Sep 18 19:08:53.828873 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:53.828846    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc"]
Sep 18 19:08:53.829434 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:08:53.829411    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104675b1_4c80_433d_b1a4_f28afa88700d.slice/crio-4628aa4e285b0444692df2c1d63fdc31822af9e65724ed6b48b87ce4ebdda97d WatchSource:0}: Error finding container 4628aa4e285b0444692df2c1d63fdc31822af9e65724ed6b48b87ce4ebdda97d: Status 404 returned error can't find the container with id 4628aa4e285b0444692df2c1d63fdc31822af9e65724ed6b48b87ce4ebdda97d
Sep 18 19:08:54.051459 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:54.051416    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Sep 18 19:08:54.051655 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:54.051486    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk" podUID="fd206272-6e62-4898-9dce-6d53c37e6d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Sep 18 19:08:54.818228 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:54.818181    2558 generic.go:356] "Generic (PLEG): container finished" podID="104675b1-4c80-433d-b1a4-f28afa88700d" containerID="c60fa1430d14a9fa96021779074e1c465d71849a60d42043849919ebbd4de608" exitCode=0
Sep 18 19:08:54.818576 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:54.818237    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc" event={"ID":"104675b1-4c80-433d-b1a4-f28afa88700d","Type":"ContainerDied","Data":"c60fa1430d14a9fa96021779074e1c465d71849a60d42043849919ebbd4de608"}
Sep 18 19:08:54.818576 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:54.818274    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc" event={"ID":"104675b1-4c80-433d-b1a4-f28afa88700d","Type":"ContainerStarted","Data":"4628aa4e285b0444692df2c1d63fdc31822af9e65724ed6b48b87ce4ebdda97d"}
Sep 18 19:08:55.055927 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:55.055846    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:55.056168 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:55.056130    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:55.056796 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:55.056778    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:08:55.961895 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:55.961872    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc"
Sep 18 19:08:56.012731 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:56.012705    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/104675b1-4c80-433d-b1a4-f28afa88700d-maas-api-service-ca\") pod \"104675b1-4c80-433d-b1a4-f28afa88700d\" (UID: \"104675b1-4c80-433d-b1a4-f28afa88700d\") "
Sep 18 19:08:56.013073 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:56.013048    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104675b1-4c80-433d-b1a4-f28afa88700d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "104675b1-4c80-433d-b1a4-f28afa88700d" (UID: "104675b1-4c80-433d-b1a4-f28afa88700d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:08:56.114340 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:56.114308    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/104675b1-4c80-433d-b1a4-f28afa88700d-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:08:56.832104 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:56.832075    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc"
Sep 18 19:08:56.832323 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:56.832120    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc" event={"ID":"104675b1-4c80-433d-b1a4-f28afa88700d","Type":"ContainerDied","Data":"4628aa4e285b0444692df2c1d63fdc31822af9e65724ed6b48b87ce4ebdda97d"}
Sep 18 19:08:56.832323 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:08:56.832170    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4628aa4e285b0444692df2c1d63fdc31822af9e65724ed6b48b87ce4ebdda97d"
Sep 18 19:09:00.153045 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:00.153007    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc"]
Sep 18 19:09:00.156585 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:00.156561    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5855d4-gqfnc"]
Sep 18 19:09:00.576566 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:00.576486    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="104675b1-4c80-433d-b1a4-f28afa88700d" path="/var/lib/kubelet/pods/104675b1-4c80-433d-b1a4-f28afa88700d/volumes"
Sep 18 19:09:02.763642 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:02.763613    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 19:09:02.765295 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:02.765270    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 19:09:02.766528 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:02.766507    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 19:09:02.768235 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:02.768214    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 19:09:03.142656 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:03.142626    2558 scope.go:117] "RemoveContainer" containerID="5e94745cd09ed17ba6c68ff09040b24c63b612b332dad067f85208a73eecfc50"
Sep 18 19:09:07.687236 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:07.687204    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"]
Sep 18 19:09:07.687701 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:07.687459    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk" podUID="fd206272-6e62-4898-9dce-6d53c37e6d6c" containerName="istio-proxy" containerID="cri-o://ada5e7c06ad6ad150681edd9d65d70f19f4b15913c9833ba5095e424e599a2e7" gracePeriod=30
Sep 18 19:09:10.059137 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:10.059100    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 19:09:10.059522 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:10.059163    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk" podUID="fd206272-6e62-4898-9dce-6d53c37e6d6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:09:11.287447 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:11.287410    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7"]
Sep 18 19:09:11.287843 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:11.287833    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104675b1-4c80-433d-b1a4-f28afa88700d" containerName="revoke-keys"
Sep 18 19:09:11.287887 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:11.287844    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="104675b1-4c80-433d-b1a4-f28afa88700d" containerName="revoke-keys"
Sep 18 19:09:11.287932 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:11.287921    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="104675b1-4c80-433d-b1a4-f28afa88700d" containerName="revoke-keys"
Sep 18 19:09:11.292438 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:11.292419    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7"
Sep 18 19:09:11.296101 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:11.295954    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:09:11.296101 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:11.295957    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 19:09:11.300278 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:11.300240    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7"]
Sep 18 19:09:11.367686 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:11.367645    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5386c1-ece5-484d-97eb-341d0f356045-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7\" (UID: \"0a5386c1-ece5-484d-97eb-341d0f356045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7"
Sep 18 19:09:11.468244 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:11.468179    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5386c1-ece5-484d-97eb-341d0f356045-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7\" (UID: \"0a5386c1-ece5-484d-97eb-341d0f356045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7"
Sep 18 19:09:11.468899 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:11.468875    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5386c1-ece5-484d-97eb-341d0f356045-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7\" (UID: \"0a5386c1-ece5-484d-97eb-341d0f356045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7"
Sep 18 19:09:11.604913 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:11.604875    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7"
Sep 18 19:09:11.878460 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:11.878328    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7"]
Sep 18 19:09:11.881416 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:09:11.881389    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5386c1_ece5_484d_97eb_341d0f356045.slice/crio-14aa7bdd687463fcaea6acff966813e5c2a79d23eab7692b5564194ec94d0209 WatchSource:0}: Error finding container 14aa7bdd687463fcaea6acff966813e5c2a79d23eab7692b5564194ec94d0209: Status 404 returned error can't find the container with id 14aa7bdd687463fcaea6acff966813e5c2a79d23eab7692b5564194ec94d0209
Sep 18 19:09:11.938709 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:11.938681    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7" event={"ID":"0a5386c1-ece5-484d-97eb-341d0f356045","Type":"ContainerStarted","Data":"14aa7bdd687463fcaea6acff966813e5c2a79d23eab7692b5564194ec94d0209"}
Sep 18 19:09:12.946102 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:12.946075    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:09:12.947860 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:12.947831    2558 generic.go:356] "Generic (PLEG): container finished" podID="fd206272-6e62-4898-9dce-6d53c37e6d6c" containerID="ada5e7c06ad6ad150681edd9d65d70f19f4b15913c9833ba5095e424e599a2e7" exitCode=0
Sep 18 19:09:12.947983 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:12.947902    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk" event={"ID":"fd206272-6e62-4898-9dce-6d53c37e6d6c","Type":"ContainerDied","Data":"ada5e7c06ad6ad150681edd9d65d70f19f4b15913c9833ba5095e424e599a2e7"}
Sep 18 19:09:12.947983 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:12.947968    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk" event={"ID":"fd206272-6e62-4898-9dce-6d53c37e6d6c","Type":"ContainerDied","Data":"ba852a5bb63e174ca437628945702898802fc727a42a0f49687dbb43dd5abac9"}
Sep 18 19:09:12.948141 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:12.947998    2558 scope.go:117] "RemoveContainer" containerID="ada5e7c06ad6ad150681edd9d65d70f19f4b15913c9833ba5095e424e599a2e7"
Sep 18 19:09:12.949877 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:12.949843    2558 generic.go:356] "Generic (PLEG): container finished" podID="0a5386c1-ece5-484d-97eb-341d0f356045" containerID="6a4207c65645ac312ac76ec767d2d6efbec8a3c6a63214b98ee7c2bda77856a9" exitCode=0
Sep 18 19:09:12.949998 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:12.949977    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7" event={"ID":"0a5386c1-ece5-484d-97eb-341d0f356045","Type":"ContainerDied","Data":"6a4207c65645ac312ac76ec767d2d6efbec8a3c6a63214b98ee7c2bda77856a9"}
Sep 18 19:09:12.958243 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:12.958223    2558 scope.go:117] "RemoveContainer" containerID="5af8f5f104920d9aa3c4ba5f4921d2f660e2fe1052f91798cecc8340ae5f9afc"
Sep 18 19:09:12.967099 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:12.967081    2558 scope.go:117] "RemoveContainer" containerID="ada5e7c06ad6ad150681edd9d65d70f19f4b15913c9833ba5095e424e599a2e7"
Sep 18 19:09:12.967402 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:09:12.967384    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada5e7c06ad6ad150681edd9d65d70f19f4b15913c9833ba5095e424e599a2e7\": container with ID starting with ada5e7c06ad6ad150681edd9d65d70f19f4b15913c9833ba5095e424e599a2e7 not found: ID does not exist" containerID="ada5e7c06ad6ad150681edd9d65d70f19f4b15913c9833ba5095e424e599a2e7"
Sep 18 19:09:12.967471 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:12.967412    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada5e7c06ad6ad150681edd9d65d70f19f4b15913c9833ba5095e424e599a2e7"} err="failed to get container status \"ada5e7c06ad6ad150681edd9d65d70f19f4b15913c9833ba5095e424e599a2e7\": rpc error: code = NotFound desc = could not find container \"ada5e7c06ad6ad150681edd9d65d70f19f4b15913c9833ba5095e424e599a2e7\": container with ID starting with ada5e7c06ad6ad150681edd9d65d70f19f4b15913c9833ba5095e424e599a2e7 not found: ID does not exist"
Sep 18 19:09:12.967471 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:12.967430    2558 scope.go:117] "RemoveContainer" containerID="5af8f5f104920d9aa3c4ba5f4921d2f660e2fe1052f91798cecc8340ae5f9afc"
Sep 18 19:09:12.967685 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:09:12.967667    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af8f5f104920d9aa3c4ba5f4921d2f660e2fe1052f91798cecc8340ae5f9afc\": container with ID starting with 5af8f5f104920d9aa3c4ba5f4921d2f660e2fe1052f91798cecc8340ae5f9afc not found: ID does not exist" containerID="5af8f5f104920d9aa3c4ba5f4921d2f660e2fe1052f91798cecc8340ae5f9afc"
Sep 18 19:09:12.967725 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:12.967691    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af8f5f104920d9aa3c4ba5f4921d2f660e2fe1052f91798cecc8340ae5f9afc"} err="failed to get container status \"5af8f5f104920d9aa3c4ba5f4921d2f660e2fe1052f91798cecc8340ae5f9afc\": rpc error: code = NotFound desc = could not find container \"5af8f5f104920d9aa3c4ba5f4921d2f660e2fe1052f91798cecc8340ae5f9afc\": container with ID starting with 5af8f5f104920d9aa3c4ba5f4921d2f660e2fe1052f91798cecc8340ae5f9afc not found: ID does not exist"
Sep 18 19:09:13.082678 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.082594    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-workload-certs\") pod \"fd206272-6e62-4898-9dce-6d53c37e6d6c\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") "
Sep 18 19:09:13.082678 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.082645    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-credential-socket\") pod \"fd206272-6e62-4898-9dce-6d53c37e6d6c\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") "
Sep 18 19:09:13.082914 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.082692    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-data\") pod \"fd206272-6e62-4898-9dce-6d53c37e6d6c\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") "
Sep 18 19:09:13.082914 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.082720    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-token\") pod \"fd206272-6e62-4898-9dce-6d53c37e6d6c\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") "
Sep 18 19:09:13.082914 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.082735    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-envoy\") pod \"fd206272-6e62-4898-9dce-6d53c37e6d6c\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") "
Sep 18 19:09:13.082914 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.082769    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-workload-socket\") pod \"fd206272-6e62-4898-9dce-6d53c37e6d6c\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") "
Sep 18 19:09:13.082914 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.082799    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd206272-6e62-4898-9dce-6d53c37e6d6c-istiod-ca-cert\") pod \"fd206272-6e62-4898-9dce-6d53c37e6d6c\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") "
Sep 18 19:09:13.082914 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.082857    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-podinfo\") pod \"fd206272-6e62-4898-9dce-6d53c37e6d6c\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") "
Sep 18 19:09:13.082914 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.082890    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4zr7\" (UniqueName: \"kubernetes.io/projected/fd206272-6e62-4898-9dce-6d53c37e6d6c-kube-api-access-s4zr7\") pod \"fd206272-6e62-4898-9dce-6d53c37e6d6c\" (UID: \"fd206272-6e62-4898-9dce-6d53c37e6d6c\") "
Sep 18 19:09:13.083289 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.082915    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fd206272-6e62-4898-9dce-6d53c37e6d6c" (UID: "fd206272-6e62-4898-9dce-6d53c37e6d6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:09:13.083289 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.083015    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fd206272-6e62-4898-9dce-6d53c37e6d6c" (UID: "fd206272-6e62-4898-9dce-6d53c37e6d6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:09:13.083289 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.083028    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-data" (OuterVolumeSpecName: "istio-data") pod "fd206272-6e62-4898-9dce-6d53c37e6d6c" (UID: "fd206272-6e62-4898-9dce-6d53c37e6d6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:09:13.083289 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.083219    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fd206272-6e62-4898-9dce-6d53c37e6d6c" (UID: "fd206272-6e62-4898-9dce-6d53c37e6d6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:09:13.083527 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.083365    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:13.083527 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.083385    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:13.083527 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.083397    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:13.083527 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.083409    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:13.083728 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.083632    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd206272-6e62-4898-9dce-6d53c37e6d6c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fd206272-6e62-4898-9dce-6d53c37e6d6c" (UID: "fd206272-6e62-4898-9dce-6d53c37e6d6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:09:13.085086 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.085056    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fd206272-6e62-4898-9dce-6d53c37e6d6c" (UID: "fd206272-6e62-4898-9dce-6d53c37e6d6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 19:09:13.085262 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.085119    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fd206272-6e62-4898-9dce-6d53c37e6d6c" (UID: "fd206272-6e62-4898-9dce-6d53c37e6d6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:09:13.085497 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.085482    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd206272-6e62-4898-9dce-6d53c37e6d6c" (UID: "fd206272-6e62-4898-9dce-6d53c37e6d6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:09:13.085535 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.085497    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd206272-6e62-4898-9dce-6d53c37e6d6c-kube-api-access-s4zr7" (OuterVolumeSpecName: "kube-api-access-s4zr7") pod "fd206272-6e62-4898-9dce-6d53c37e6d6c" (UID: "fd206272-6e62-4898-9dce-6d53c37e6d6c"). InnerVolumeSpecName "kube-api-access-s4zr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:09:13.184660 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.184631    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:13.184660 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.184654    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:13.184660 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.184662    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd206272-6e62-4898-9dce-6d53c37e6d6c-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:13.184893 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.184671    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd206272-6e62-4898-9dce-6d53c37e6d6c-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:13.184893 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.184680    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4zr7\" (UniqueName: \"kubernetes.io/projected/fd206272-6e62-4898-9dce-6d53c37e6d6c-kube-api-access-s4zr7\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:13.957488 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.957458    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"
Sep 18 19:09:13.986333 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.984575    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"]
Sep 18 19:09:13.987384 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:13.987339    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5855d4-openshift-default-985786f94-bj8hk"]
Sep 18 19:09:14.100934 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:14.100913    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7"
Sep 18 19:09:14.194688 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:14.194648    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5386c1-ece5-484d-97eb-341d0f356045-maas-api-service-ca\") pod \"0a5386c1-ece5-484d-97eb-341d0f356045\" (UID: \"0a5386c1-ece5-484d-97eb-341d0f356045\") "
Sep 18 19:09:14.195066 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:14.195032    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5386c1-ece5-484d-97eb-341d0f356045-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a5386c1-ece5-484d-97eb-341d0f356045" (UID: "0a5386c1-ece5-484d-97eb-341d0f356045"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:09:14.296114 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:14.296029    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a5386c1-ece5-484d-97eb-341d0f356045-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:14.575718 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:14.575633    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd206272-6e62-4898-9dce-6d53c37e6d6c" path="/var/lib/kubelet/pods/fd206272-6e62-4898-9dce-6d53c37e6d6c/volumes"
Sep 18 19:09:14.965657 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:14.965623    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7"
Sep 18 19:09:14.965657 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:14.965650    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7" event={"ID":"0a5386c1-ece5-484d-97eb-341d0f356045","Type":"ContainerDied","Data":"14aa7bdd687463fcaea6acff966813e5c2a79d23eab7692b5564194ec94d0209"}
Sep 18 19:09:14.966239 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:14.965700    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14aa7bdd687463fcaea6acff966813e5c2a79d23eab7692b5564194ec94d0209"
Sep 18 19:09:27.746623 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:27.746586    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7"]
Sep 18 19:09:27.758515 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:27.758475    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0991d8-969q7"]
Sep 18 19:09:28.576156 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:28.576120    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a5386c1-ece5-484d-97eb-341d0f356045" path="/var/lib/kubelet/pods/0a5386c1-ece5-484d-97eb-341d0f356045/volumes"
Sep 18 19:09:33.111668 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:33.111630    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"]
Sep 18 19:09:48.146693 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.146641    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd" podUID="a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee" containerName="main" containerID="cri-o://2edf0fa09fb91f759cc7789cbd55c005644ce75179c04707e56017a06a1383d3" gracePeriod=45
Sep 18 19:09:48.401354 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.401294    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:09:48.531489 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.531458    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-dshm\") pod \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") "
Sep 18 19:09:48.531489 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.531494    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-tls-certs\") pod \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") "
Sep 18 19:09:48.531732 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.531525    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcbrz\" (UniqueName: \"kubernetes.io/projected/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-kube-api-access-qcbrz\") pod \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") "
Sep 18 19:09:48.531732 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.531553    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-tmp-dir\") pod \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") "
Sep 18 19:09:48.531732 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.531666    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-model-cache\") pod \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") "
Sep 18 19:09:48.531887 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.531749    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-home\") pod \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\" (UID: \"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee\") "
Sep 18 19:09:48.531947 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.531919    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee" (UID: "a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:09:48.532001 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.531976    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee" (UID: "a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:09:48.532137 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.532107    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-home" (OuterVolumeSpecName: "home") pod "a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee" (UID: "a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:09:48.532209 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.532146    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-tmp-dir\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:48.532209 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.532168    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-model-cache\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:48.533765 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.533737    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee" (UID: "a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:09:48.533873 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.533792    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-dshm" (OuterVolumeSpecName: "dshm") pod "a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee" (UID: "a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:09:48.533873 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.533833    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-kube-api-access-qcbrz" (OuterVolumeSpecName: "kube-api-access-qcbrz") pod "a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee" (UID: "a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee"). InnerVolumeSpecName "kube-api-access-qcbrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:09:48.633673 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.633637    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-home\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:48.633673 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.633665    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-dshm\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:48.633673 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.633676    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-tls-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:48.633905 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:48.633690    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcbrz\" (UniqueName: \"kubernetes.io/projected/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee-kube-api-access-qcbrz\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:09:49.182318 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:49.182281    2558 generic.go:356] "Generic (PLEG): container finished" podID="a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee" containerID="2edf0fa09fb91f759cc7789cbd55c005644ce75179c04707e56017a06a1383d3" exitCode=0
Sep 18 19:09:49.182759 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:49.182327    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"
Sep 18 19:09:49.182759 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:49.182364    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd" event={"ID":"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee","Type":"ContainerDied","Data":"2edf0fa09fb91f759cc7789cbd55c005644ce75179c04707e56017a06a1383d3"}
Sep 18 19:09:49.182759 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:49.182416    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd" event={"ID":"a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee","Type":"ContainerDied","Data":"386df5e33555e53e3ad042ec557983fae9ff16b4a3ba786c7d2d1b16ffd56c09"}
Sep 18 19:09:49.182759 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:49.182435    2558 scope.go:117] "RemoveContainer" containerID="2edf0fa09fb91f759cc7789cbd55c005644ce75179c04707e56017a06a1383d3"
Sep 18 19:09:49.191658 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:49.191638    2558 scope.go:117] "RemoveContainer" containerID="2edf0fa09fb91f759cc7789cbd55c005644ce75179c04707e56017a06a1383d3"
Sep 18 19:09:49.191919 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:09:49.191898    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2edf0fa09fb91f759cc7789cbd55c005644ce75179c04707e56017a06a1383d3\": container with ID starting with 2edf0fa09fb91f759cc7789cbd55c005644ce75179c04707e56017a06a1383d3 not found: ID does not exist" containerID="2edf0fa09fb91f759cc7789cbd55c005644ce75179c04707e56017a06a1383d3"
Sep 18 19:09:49.191989 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:49.191930    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2edf0fa09fb91f759cc7789cbd55c005644ce75179c04707e56017a06a1383d3"} err="failed to get container status \"2edf0fa09fb91f759cc7789cbd55c005644ce75179c04707e56017a06a1383d3\": rpc error: code = NotFound desc = could not find container \"2edf0fa09fb91f759cc7789cbd55c005644ce75179c04707e56017a06a1383d3\": container with ID starting with 2edf0fa09fb91f759cc7789cbd55c005644ce75179c04707e56017a06a1383d3 not found: ID does not exist"
Sep 18 19:09:49.199073 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:49.199051    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"]
Sep 18 19:09:49.203153 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:49.203127    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-0991d8/ipp-route-0991d8-kserve-59c7fdc78c-rsxpd"]
Sep 18 19:09:50.575844 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:09:50.575813    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee" path="/var/lib/kubelet/pods/a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee/volumes"
Sep 18 19:10:16.455836 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.455702    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"]
Sep 18 19:10:16.456795 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.456681    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2" podUID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" containerName="istio-proxy" containerID="cri-o://571afc33ae9905aabbcd8754b0ed4e9f6c83730795c2b04ffd890f9a5b7591cc" gracePeriod=30
Sep 18 19:10:16.816694 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.816609    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk"]
Sep 18 19:10:16.817017 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.817004    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a5386c1-ece5-484d-97eb-341d0f356045" containerName="revoke-keys"
Sep 18 19:10:16.817065 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.817018    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a5386c1-ece5-484d-97eb-341d0f356045" containerName="revoke-keys"
Sep 18 19:10:16.817065 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.817038    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd206272-6e62-4898-9dce-6d53c37e6d6c" containerName="istio-proxy"
Sep 18 19:10:16.817065 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.817044    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd206272-6e62-4898-9dce-6d53c37e6d6c" containerName="istio-proxy"
Sep 18 19:10:16.817065 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.817051    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee" containerName="main"
Sep 18 19:10:16.817065 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.817057    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee" containerName="main"
Sep 18 19:10:16.817246 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.817067    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd206272-6e62-4898-9dce-6d53c37e6d6c" containerName="istio-proxy"
Sep 18 19:10:16.817246 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.817073    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd206272-6e62-4898-9dce-6d53c37e6d6c" containerName="istio-proxy"
Sep 18 19:10:16.817246 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.817132    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd206272-6e62-4898-9dce-6d53c37e6d6c" containerName="istio-proxy"
Sep 18 19:10:16.817246 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.817141    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd206272-6e62-4898-9dce-6d53c37e6d6c" containerName="istio-proxy"
Sep 18 19:10:16.817246 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.817148    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a5386c1-ece5-484d-97eb-341d0f356045" containerName="revoke-keys"
Sep 18 19:10:16.817246 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.817155    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3a5b59c-bc56-4149-9eb8-bcc071e5e2ee" containerName="main"
Sep 18 19:10:16.821731 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.821714    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk"
Sep 18 19:10:16.824487 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.824468    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:10:16.825090 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.825070    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 19:10:16.827852 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.827830    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk"]
Sep 18 19:10:16.904707 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:16.904674    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381ffa5a-956a-47fe-bf82-b83d26a98014-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk\" (UID: \"381ffa5a-956a-47fe-bf82-b83d26a98014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk"
Sep 18 19:10:17.006058 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:17.006019    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381ffa5a-956a-47fe-bf82-b83d26a98014-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk\" (UID: \"381ffa5a-956a-47fe-bf82-b83d26a98014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk"
Sep 18 19:10:17.006687 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:17.006667    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381ffa5a-956a-47fe-bf82-b83d26a98014-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk\" (UID: \"381ffa5a-956a-47fe-bf82-b83d26a98014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk"
Sep 18 19:10:17.134274 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:17.134146    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk"
Sep 18 19:10:17.411110 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:17.411086    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk"]
Sep 18 19:10:17.413551 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:10:17.413520    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381ffa5a_956a_47fe_bf82_b83d26a98014.slice/crio-995adf3bfa84a7a4aabb0b527ca906812b092eca9c64435c6da76f0fd693bc6c WatchSource:0}: Error finding container 995adf3bfa84a7a4aabb0b527ca906812b092eca9c64435c6da76f0fd693bc6c: Status 404 returned error can't find the container with id 995adf3bfa84a7a4aabb0b527ca906812b092eca9c64435c6da76f0fd693bc6c
Sep 18 19:10:18.360961 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:18.360923    2558 generic.go:356] "Generic (PLEG): container finished" podID="381ffa5a-956a-47fe-bf82-b83d26a98014" containerID="405964b1d0d3319dfc506c58efec012d94c0af7da58a5bc8c76c36dafeeeb6d8" exitCode=0
Sep 18 19:10:18.361377 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:18.361009    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk" event={"ID":"381ffa5a-956a-47fe-bf82-b83d26a98014","Type":"ContainerDied","Data":"405964b1d0d3319dfc506c58efec012d94c0af7da58a5bc8c76c36dafeeeb6d8"}
Sep 18 19:10:18.361377 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:18.361057    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk" event={"ID":"381ffa5a-956a-47fe-bf82-b83d26a98014","Type":"ContainerStarted","Data":"995adf3bfa84a7a4aabb0b527ca906812b092eca9c64435c6da76f0fd693bc6c"}
Sep 18 19:10:19.501981 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:19.501959    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk"
Sep 18 19:10:19.630785 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:19.630702    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381ffa5a-956a-47fe-bf82-b83d26a98014-maas-api-service-ca\") pod \"381ffa5a-956a-47fe-bf82-b83d26a98014\" (UID: \"381ffa5a-956a-47fe-bf82-b83d26a98014\") "
Sep 18 19:10:19.631089 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:19.631067    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381ffa5a-956a-47fe-bf82-b83d26a98014-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "381ffa5a-956a-47fe-bf82-b83d26a98014" (UID: "381ffa5a-956a-47fe-bf82-b83d26a98014"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:10:19.732100 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:19.732069    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381ffa5a-956a-47fe-bf82-b83d26a98014-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:20.374635 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:20.374604    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk"
Sep 18 19:10:20.374635 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:20.374612    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk" event={"ID":"381ffa5a-956a-47fe-bf82-b83d26a98014","Type":"ContainerDied","Data":"995adf3bfa84a7a4aabb0b527ca906812b092eca9c64435c6da76f0fd693bc6c"}
Sep 18 19:10:20.374857 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:20.374659    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="995adf3bfa84a7a4aabb0b527ca906812b092eca9c64435c6da76f0fd693bc6c"
Sep 18 19:10:21.717822 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.717793    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:10:21.851490 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.851455    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-envoy\") pod \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") "
Sep 18 19:10:21.851664 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.851507    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-data\") pod \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") "
Sep 18 19:10:21.851664 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.851527    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-credential-socket\") pod \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") "
Sep 18 19:10:21.851664 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.851563    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-token\") pod \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") "
Sep 18 19:10:21.851664 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.851616    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-podinfo\") pod \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") "
Sep 18 19:10:21.851664 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.851637    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-workload-socket\") pod \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") "
Sep 18 19:10:21.851927 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.851674    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istiod-ca-cert\") pod \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") "
Sep 18 19:10:21.851927 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.851806    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-workload-certs\") pod \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") "
Sep 18 19:10:21.851927 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.851821    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-data" (OuterVolumeSpecName: "istio-data") pod "4e4e8663-2d6b-4494-90d2-34a56f47c4aa" (UID: "4e4e8663-2d6b-4494-90d2-34a56f47c4aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:10:21.851927 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.851852    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcsgr\" (UniqueName: \"kubernetes.io/projected/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-kube-api-access-pcsgr\") pod \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\" (UID: \"4e4e8663-2d6b-4494-90d2-34a56f47c4aa\") "
Sep 18 19:10:21.851927 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.851913    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4e4e8663-2d6b-4494-90d2-34a56f47c4aa" (UID: "4e4e8663-2d6b-4494-90d2-34a56f47c4aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:10:21.852353 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.852011    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4e4e8663-2d6b-4494-90d2-34a56f47c4aa" (UID: "4e4e8663-2d6b-4494-90d2-34a56f47c4aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:10:21.852353 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.852084    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4e4e8663-2d6b-4494-90d2-34a56f47c4aa" (UID: "4e4e8663-2d6b-4494-90d2-34a56f47c4aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:10:21.852353 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.852138    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4e4e8663-2d6b-4494-90d2-34a56f47c4aa" (UID: "4e4e8663-2d6b-4494-90d2-34a56f47c4aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:10:21.852509 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.852376    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:21.852509 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.852398    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:21.852509 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.852412    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:21.852509 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.852424    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:21.852509 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.852439    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:21.853937 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.853910    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "4e4e8663-2d6b-4494-90d2-34a56f47c4aa" (UID: "4e4e8663-2d6b-4494-90d2-34a56f47c4aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:10:21.854110 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.854083    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-kube-api-access-pcsgr" (OuterVolumeSpecName: "kube-api-access-pcsgr") pod "4e4e8663-2d6b-4494-90d2-34a56f47c4aa" (UID: "4e4e8663-2d6b-4494-90d2-34a56f47c4aa"). InnerVolumeSpecName "kube-api-access-pcsgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:10:21.854219 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.854161    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4e4e8663-2d6b-4494-90d2-34a56f47c4aa" (UID: "4e4e8663-2d6b-4494-90d2-34a56f47c4aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 19:10:21.854281 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.854258    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4e4e8663-2d6b-4494-90d2-34a56f47c4aa" (UID: "4e4e8663-2d6b-4494-90d2-34a56f47c4aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:10:21.953970 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.953919    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:21.953970 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.953959    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:21.953970 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.953967    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:21.953970 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:21.953976    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcsgr\" (UniqueName: \"kubernetes.io/projected/4e4e8663-2d6b-4494-90d2-34a56f47c4aa-kube-api-access-pcsgr\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:22.388357 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:22.388322    2558 generic.go:356] "Generic (PLEG): container finished" podID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" containerID="571afc33ae9905aabbcd8754b0ed4e9f6c83730795c2b04ffd890f9a5b7591cc" exitCode=0
Sep 18 19:10:22.388529 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:22.388364    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2" event={"ID":"4e4e8663-2d6b-4494-90d2-34a56f47c4aa","Type":"ContainerDied","Data":"571afc33ae9905aabbcd8754b0ed4e9f6c83730795c2b04ffd890f9a5b7591cc"}
Sep 18 19:10:22.388529 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:22.388373    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"
Sep 18 19:10:22.388529 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:22.388435    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2" event={"ID":"4e4e8663-2d6b-4494-90d2-34a56f47c4aa","Type":"ContainerDied","Data":"e3c1b98a75a08668df8d24d3c653e5f40a1bfe5b80827fe8fa7a454477d009f8"}
Sep 18 19:10:22.388529 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:22.388459    2558 scope.go:117] "RemoveContainer" containerID="571afc33ae9905aabbcd8754b0ed4e9f6c83730795c2b04ffd890f9a5b7591cc"
Sep 18 19:10:22.399047 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:22.399029    2558 scope.go:117] "RemoveContainer" containerID="0eb1a990905ed76eb28d3f003ab81acd11ccec57567e05717e3a6c9635101409"
Sep 18 19:10:22.407734 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:22.407712    2558 scope.go:117] "RemoveContainer" containerID="571afc33ae9905aabbcd8754b0ed4e9f6c83730795c2b04ffd890f9a5b7591cc"
Sep 18 19:10:22.408304 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:10:22.408264    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571afc33ae9905aabbcd8754b0ed4e9f6c83730795c2b04ffd890f9a5b7591cc\": container with ID starting with 571afc33ae9905aabbcd8754b0ed4e9f6c83730795c2b04ffd890f9a5b7591cc not found: ID does not exist" containerID="571afc33ae9905aabbcd8754b0ed4e9f6c83730795c2b04ffd890f9a5b7591cc"
Sep 18 19:10:22.408404 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:22.408308    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571afc33ae9905aabbcd8754b0ed4e9f6c83730795c2b04ffd890f9a5b7591cc"} err="failed to get container status \"571afc33ae9905aabbcd8754b0ed4e9f6c83730795c2b04ffd890f9a5b7591cc\": rpc error: code = NotFound desc = could not find container \"571afc33ae9905aabbcd8754b0ed4e9f6c83730795c2b04ffd890f9a5b7591cc\": container with ID starting with 571afc33ae9905aabbcd8754b0ed4e9f6c83730795c2b04ffd890f9a5b7591cc not found: ID does not exist"
Sep 18 19:10:22.408404 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:22.408332    2558 scope.go:117] "RemoveContainer" containerID="0eb1a990905ed76eb28d3f003ab81acd11ccec57567e05717e3a6c9635101409"
Sep 18 19:10:22.408718 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:10:22.408697    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb1a990905ed76eb28d3f003ab81acd11ccec57567e05717e3a6c9635101409\": container with ID starting with 0eb1a990905ed76eb28d3f003ab81acd11ccec57567e05717e3a6c9635101409 not found: ID does not exist" containerID="0eb1a990905ed76eb28d3f003ab81acd11ccec57567e05717e3a6c9635101409"
Sep 18 19:10:22.408780 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:22.408727    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb1a990905ed76eb28d3f003ab81acd11ccec57567e05717e3a6c9635101409"} err="failed to get container status \"0eb1a990905ed76eb28d3f003ab81acd11ccec57567e05717e3a6c9635101409\": rpc error: code = NotFound desc = could not find container \"0eb1a990905ed76eb28d3f003ab81acd11ccec57567e05717e3a6c9635101409\": container with ID starting with 0eb1a990905ed76eb28d3f003ab81acd11ccec57567e05717e3a6c9635101409 not found: ID does not exist"
Sep 18 19:10:22.411350 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:22.411329    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"]
Sep 18 19:10:22.415032 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:22.415011    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0991d8-openshift-default-59cfc6dfbd-4wwp2"]
Sep 18 19:10:22.575543 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:22.575509    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" path="/var/lib/kubelet/pods/4e4e8663-2d6b-4494-90d2-34a56f47c4aa/volumes"
Sep 18 19:10:27.738830 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:27.738793    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk"]
Sep 18 19:10:27.746436 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:27.746408    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d6246c-czkpk"]
Sep 18 19:10:28.575470 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:28.575434    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="381ffa5a-956a-47fe-bf82-b83d26a98014" path="/var/lib/kubelet/pods/381ffa5a-956a-47fe-bf82-b83d26a98014/volumes"
Sep 18 19:10:34.858154 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:34.858115    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"]
Sep 18 19:10:34.858603 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:34.858338    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9" podUID="c01ed0fa-e16d-4119-893b-24d93d5f70f5" containerName="istio-proxy" containerID="cri-o://0258bc7637bf4e368eca082497e2ac6009e858ae9975f3fc9db66afe5a79a658" gracePeriod=30
Sep 18 19:10:35.204504 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.204425    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw"]
Sep 18 19:10:35.204826 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.204812    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381ffa5a-956a-47fe-bf82-b83d26a98014" containerName="revoke-keys"
Sep 18 19:10:35.204869 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.204828    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="381ffa5a-956a-47fe-bf82-b83d26a98014" containerName="revoke-keys"
Sep 18 19:10:35.204869 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.204840    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" containerName="istio-proxy"
Sep 18 19:10:35.204869 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.204846    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" containerName="istio-proxy"
Sep 18 19:10:35.204869 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.204863    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" containerName="istio-proxy"
Sep 18 19:10:35.204869 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.204869    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" containerName="istio-proxy"
Sep 18 19:10:35.205026 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.204944    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" containerName="istio-proxy"
Sep 18 19:10:35.205026 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.204952    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e4e8663-2d6b-4494-90d2-34a56f47c4aa" containerName="istio-proxy"
Sep 18 19:10:35.205026 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.204959    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="381ffa5a-956a-47fe-bf82-b83d26a98014" containerName="revoke-keys"
Sep 18 19:10:35.208263 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.208246    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw"
Sep 18 19:10:35.210667 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.210647    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:10:35.210859 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.210845    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 19:10:35.218043 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.218016    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw"]
Sep 18 19:10:35.382166 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.382129    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6750e8fa-11d6-4ffe-82ab-b9ca940a62c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw\" (UID: \"6750e8fa-11d6-4ffe-82ab-b9ca940a62c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw"
Sep 18 19:10:35.483315 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.483226    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6750e8fa-11d6-4ffe-82ab-b9ca940a62c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw\" (UID: \"6750e8fa-11d6-4ffe-82ab-b9ca940a62c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw"
Sep 18 19:10:35.483853 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.483833    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6750e8fa-11d6-4ffe-82ab-b9ca940a62c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw\" (UID: \"6750e8fa-11d6-4ffe-82ab-b9ca940a62c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw"
Sep 18 19:10:35.520205 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.520157    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw"
Sep 18 19:10:35.805268 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:35.805235    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw"]
Sep 18 19:10:35.806631 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:10:35.806605    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6750e8fa_11d6_4ffe_82ab_b9ca940a62c8.slice/crio-8399e450bfa94517bc996339ec7b5546045a65fa60e84a5f6d0b4b9e1aa21688 WatchSource:0}: Error finding container 8399e450bfa94517bc996339ec7b5546045a65fa60e84a5f6d0b4b9e1aa21688: Status 404 returned error can't find the container with id 8399e450bfa94517bc996339ec7b5546045a65fa60e84a5f6d0b4b9e1aa21688
Sep 18 19:10:36.481258 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:36.481220    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw" event={"ID":"6750e8fa-11d6-4ffe-82ab-b9ca940a62c8","Type":"ContainerStarted","Data":"f1ddffa4ba53426284027ccba7c403e79b595ebfff31bf035b568f72caaf7e44"}
Sep 18 19:10:36.481660 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:36.481287    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw" event={"ID":"6750e8fa-11d6-4ffe-82ab-b9ca940a62c8","Type":"ContainerStarted","Data":"8399e450bfa94517bc996339ec7b5546045a65fa60e84a5f6d0b4b9e1aa21688"}
Sep 18 19:10:36.498405 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:36.498353    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw" podStartSLOduration=1.498337247 podStartE2EDuration="1.498337247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:10:36.496939447 +0000 UTC m=+1895.933518291" watchObservedRunningTime="2026-09-18 19:10:36.498337247 +0000 UTC m=+1895.934916088"
Sep 18 19:10:37.489700 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:37.489659    2558 generic.go:356] "Generic (PLEG): container finished" podID="6750e8fa-11d6-4ffe-82ab-b9ca940a62c8" containerID="f1ddffa4ba53426284027ccba7c403e79b595ebfff31bf035b568f72caaf7e44" exitCode=0
Sep 18 19:10:37.490165 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:37.489738    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw" event={"ID":"6750e8fa-11d6-4ffe-82ab-b9ca940a62c8","Type":"ContainerDied","Data":"f1ddffa4ba53426284027ccba7c403e79b595ebfff31bf035b568f72caaf7e44"}
Sep 18 19:10:38.639482 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:38.639457    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw"
Sep 18 19:10:38.816117 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:38.816035    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6750e8fa-11d6-4ffe-82ab-b9ca940a62c8-maas-api-service-ca\") pod \"6750e8fa-11d6-4ffe-82ab-b9ca940a62c8\" (UID: \"6750e8fa-11d6-4ffe-82ab-b9ca940a62c8\") "
Sep 18 19:10:38.816412 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:38.816390    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6750e8fa-11d6-4ffe-82ab-b9ca940a62c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6750e8fa-11d6-4ffe-82ab-b9ca940a62c8" (UID: "6750e8fa-11d6-4ffe-82ab-b9ca940a62c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:10:38.917280 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:38.917244    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6750e8fa-11d6-4ffe-82ab-b9ca940a62c8-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:39.504614 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:39.504575    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw" event={"ID":"6750e8fa-11d6-4ffe-82ab-b9ca940a62c8","Type":"ContainerDied","Data":"8399e450bfa94517bc996339ec7b5546045a65fa60e84a5f6d0b4b9e1aa21688"}
Sep 18 19:10:39.504614 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:39.504611    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw"
Sep 18 19:10:39.504839 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:39.504652    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8399e450bfa94517bc996339ec7b5546045a65fa60e84a5f6d0b4b9e1aa21688"
Sep 18 19:10:40.109727 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.109704    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:10:40.229906 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.229826    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-envoy\") pod \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") "
Sep 18 19:10:40.229906 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.229863    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-workload-certs\") pod \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") "
Sep 18 19:10:40.229906 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.229885    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-credential-socket\") pod \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") "
Sep 18 19:10:40.230160 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.229931    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-podinfo\") pod \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") "
Sep 18 19:10:40.230160 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.229953    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-workload-socket\") pod \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") "
Sep 18 19:10:40.230160 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.229974    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvpmd\" (UniqueName: \"kubernetes.io/projected/c01ed0fa-e16d-4119-893b-24d93d5f70f5-kube-api-access-qvpmd\") pod \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") "
Sep 18 19:10:40.230160 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.230045    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-token\") pod \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") "
Sep 18 19:10:40.230160 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.230079    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istiod-ca-cert\") pod \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") "
Sep 18 19:10:40.230466 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.230166    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-data\") pod \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\" (UID: \"c01ed0fa-e16d-4119-893b-24d93d5f70f5\") "
Sep 18 19:10:40.230466 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.230157    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c01ed0fa-e16d-4119-893b-24d93d5f70f5" (UID: "c01ed0fa-e16d-4119-893b-24d93d5f70f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:10:40.230466 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.230407    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c01ed0fa-e16d-4119-893b-24d93d5f70f5" (UID: "c01ed0fa-e16d-4119-893b-24d93d5f70f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:10:40.230466 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.230418    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c01ed0fa-e16d-4119-893b-24d93d5f70f5" (UID: "c01ed0fa-e16d-4119-893b-24d93d5f70f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:10:40.230689 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.230616    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "c01ed0fa-e16d-4119-893b-24d93d5f70f5" (UID: "c01ed0fa-e16d-4119-893b-24d93d5f70f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:10:40.230689 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.230620    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:40.230689 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.230667    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:40.230689 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.230683    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:40.230898 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.230710    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c01ed0fa-e16d-4119-893b-24d93d5f70f5" (UID: "c01ed0fa-e16d-4119-893b-24d93d5f70f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:10:40.232496 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.232460    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c01ed0fa-e16d-4119-893b-24d93d5f70f5" (UID: "c01ed0fa-e16d-4119-893b-24d93d5f70f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 19:10:40.232734 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.232703    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c01ed0fa-e16d-4119-893b-24d93d5f70f5" (UID: "c01ed0fa-e16d-4119-893b-24d93d5f70f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:10:40.232845 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.232767    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01ed0fa-e16d-4119-893b-24d93d5f70f5-kube-api-access-qvpmd" (OuterVolumeSpecName: "kube-api-access-qvpmd") pod "c01ed0fa-e16d-4119-893b-24d93d5f70f5" (UID: "c01ed0fa-e16d-4119-893b-24d93d5f70f5"). InnerVolumeSpecName "kube-api-access-qvpmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:10:40.232845 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.232778    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "c01ed0fa-e16d-4119-893b-24d93d5f70f5" (UID: "c01ed0fa-e16d-4119-893b-24d93d5f70f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:10:40.331422 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.331391    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:40.331422 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.331415    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:40.331422 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.331424    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:40.331422 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.331432    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:40.331700 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.331440    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01ed0fa-e16d-4119-893b-24d93d5f70f5-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:40.331700 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.331449    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvpmd\" (UniqueName: \"kubernetes.io/projected/c01ed0fa-e16d-4119-893b-24d93d5f70f5-kube-api-access-qvpmd\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:40.512764 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.512676    2558 generic.go:356] "Generic (PLEG): container finished" podID="c01ed0fa-e16d-4119-893b-24d93d5f70f5" containerID="0258bc7637bf4e368eca082497e2ac6009e858ae9975f3fc9db66afe5a79a658" exitCode=0
Sep 18 19:10:40.512764 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.512727    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"
Sep 18 19:10:40.512764 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.512744    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9" event={"ID":"c01ed0fa-e16d-4119-893b-24d93d5f70f5","Type":"ContainerDied","Data":"0258bc7637bf4e368eca082497e2ac6009e858ae9975f3fc9db66afe5a79a658"}
Sep 18 19:10:40.513001 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.512795    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9" event={"ID":"c01ed0fa-e16d-4119-893b-24d93d5f70f5","Type":"ContainerDied","Data":"c25ae67136f32365d93bb078df4cb6ed3e1d82cfb2b13309381aaf1104b66ed1"}
Sep 18 19:10:40.513001 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.512817    2558 scope.go:117] "RemoveContainer" containerID="0258bc7637bf4e368eca082497e2ac6009e858ae9975f3fc9db66afe5a79a658"
Sep 18 19:10:40.523129 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.523110    2558 scope.go:117] "RemoveContainer" containerID="b6af462059927b38a4b419d28fcff9dc833dc1cd31e177404bad7e28f68ae3fb"
Sep 18 19:10:40.532004 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.531983    2558 scope.go:117] "RemoveContainer" containerID="0258bc7637bf4e368eca082497e2ac6009e858ae9975f3fc9db66afe5a79a658"
Sep 18 19:10:40.532298 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:10:40.532277    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0258bc7637bf4e368eca082497e2ac6009e858ae9975f3fc9db66afe5a79a658\": container with ID starting with 0258bc7637bf4e368eca082497e2ac6009e858ae9975f3fc9db66afe5a79a658 not found: ID does not exist" containerID="0258bc7637bf4e368eca082497e2ac6009e858ae9975f3fc9db66afe5a79a658"
Sep 18 19:10:40.532360 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.532308    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0258bc7637bf4e368eca082497e2ac6009e858ae9975f3fc9db66afe5a79a658"} err="failed to get container status \"0258bc7637bf4e368eca082497e2ac6009e858ae9975f3fc9db66afe5a79a658\": rpc error: code = NotFound desc = could not find container \"0258bc7637bf4e368eca082497e2ac6009e858ae9975f3fc9db66afe5a79a658\": container with ID starting with 0258bc7637bf4e368eca082497e2ac6009e858ae9975f3fc9db66afe5a79a658 not found: ID does not exist"
Sep 18 19:10:40.532360 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.532325    2558 scope.go:117] "RemoveContainer" containerID="b6af462059927b38a4b419d28fcff9dc833dc1cd31e177404bad7e28f68ae3fb"
Sep 18 19:10:40.532595 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:10:40.532573    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6af462059927b38a4b419d28fcff9dc833dc1cd31e177404bad7e28f68ae3fb\": container with ID starting with b6af462059927b38a4b419d28fcff9dc833dc1cd31e177404bad7e28f68ae3fb not found: ID does not exist" containerID="b6af462059927b38a4b419d28fcff9dc833dc1cd31e177404bad7e28f68ae3fb"
Sep 18 19:10:40.532641 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.532604    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6af462059927b38a4b419d28fcff9dc833dc1cd31e177404bad7e28f68ae3fb"} err="failed to get container status \"b6af462059927b38a4b419d28fcff9dc833dc1cd31e177404bad7e28f68ae3fb\": rpc error: code = NotFound desc = could not find container \"b6af462059927b38a4b419d28fcff9dc833dc1cd31e177404bad7e28f68ae3fb\": container with ID starting with b6af462059927b38a4b419d28fcff9dc833dc1cd31e177404bad7e28f68ae3fb not found: ID does not exist"
Sep 18 19:10:40.536065 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.536044    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"]
Sep 18 19:10:40.539446 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.539424    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d6246c-openshift-default-77ffd47cd-wcqs9"]
Sep 18 19:10:40.576969 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:40.576939    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c01ed0fa-e16d-4119-893b-24d93d5f70f5" path="/var/lib/kubelet/pods/c01ed0fa-e16d-4119-893b-24d93d5f70f5/volumes"
Sep 18 19:10:45.806378 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:45.806327    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-298125-29829300-ctd75"]
Sep 18 19:10:45.808708 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:45.808685    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-298125-29829300-ctd75"]
Sep 18 19:10:46.025576 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:46.025533    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw"]
Sep 18 19:10:46.027918 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:46.027892    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-298125-q6sqw"]
Sep 18 19:10:46.575258 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:46.575226    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6750e8fa-11d6-4ffe-82ab-b9ca940a62c8" path="/var/lib/kubelet/pods/6750e8fa-11d6-4ffe-82ab-b9ca940a62c8/volumes"
Sep 18 19:10:46.575563 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:46.575549    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88696862-bcce-4ac3-b840-a541420b28a6" path="/var/lib/kubelet/pods/88696862-bcce-4ac3-b840-a541420b28a6/volumes"
Sep 18 19:10:52.997506 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:52.997468    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf"]
Sep 18 19:10:52.997986 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:52.997899    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01ed0fa-e16d-4119-893b-24d93d5f70f5" containerName="istio-proxy"
Sep 18 19:10:52.997986 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:52.997911    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01ed0fa-e16d-4119-893b-24d93d5f70f5" containerName="istio-proxy"
Sep 18 19:10:52.997986 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:52.997922    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6750e8fa-11d6-4ffe-82ab-b9ca940a62c8" containerName="revoke-keys"
Sep 18 19:10:52.997986 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:52.997927    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6750e8fa-11d6-4ffe-82ab-b9ca940a62c8" containerName="revoke-keys"
Sep 18 19:10:52.997986 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:52.997937    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01ed0fa-e16d-4119-893b-24d93d5f70f5" containerName="istio-proxy"
Sep 18 19:10:52.997986 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:52.997943    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01ed0fa-e16d-4119-893b-24d93d5f70f5" containerName="istio-proxy"
Sep 18 19:10:52.998179 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:52.998011    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01ed0fa-e16d-4119-893b-24d93d5f70f5" containerName="istio-proxy"
Sep 18 19:10:52.998179 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:52.998021    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01ed0fa-e16d-4119-893b-24d93d5f70f5" containerName="istio-proxy"
Sep 18 19:10:52.998179 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:52.998029    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6750e8fa-11d6-4ffe-82ab-b9ca940a62c8" containerName="revoke-keys"
Sep 18 19:10:53.002796 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:53.002778    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf"
Sep 18 19:10:53.005140 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:53.005119    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:10:53.005253 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:53.005161    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xvbnc\""
Sep 18 19:10:53.009377 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:53.009352    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf"]
Sep 18 19:10:53.044675 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:53.044646    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea08ae6-8523-4108-b93e-e772d930bf75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf\" (UID: \"3ea08ae6-8523-4108-b93e-e772d930bf75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf"
Sep 18 19:10:53.145305 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:53.145270    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea08ae6-8523-4108-b93e-e772d930bf75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf\" (UID: \"3ea08ae6-8523-4108-b93e-e772d930bf75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf"
Sep 18 19:10:53.145879 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:53.145860    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea08ae6-8523-4108-b93e-e772d930bf75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf\" (UID: \"3ea08ae6-8523-4108-b93e-e772d930bf75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf"
Sep 18 19:10:53.314461 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:53.314377    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf"
Sep 18 19:10:53.595901 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:53.595874    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf"]
Sep 18 19:10:53.597518 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:10:53.597485    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea08ae6_8523_4108_b93e_e772d930bf75.slice/crio-823185baf39e01fbb3b994d2f4b4c5814893d42a1520f35606eb52ce30671bc4 WatchSource:0}: Error finding container 823185baf39e01fbb3b994d2f4b4c5814893d42a1520f35606eb52ce30671bc4: Status 404 returned error can't find the container with id 823185baf39e01fbb3b994d2f4b4c5814893d42a1520f35606eb52ce30671bc4
Sep 18 19:10:54.611751 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:54.611715    2558 generic.go:356] "Generic (PLEG): container finished" podID="3ea08ae6-8523-4108-b93e-e772d930bf75" containerID="686b3b55e865eb78c50f60ebcfde254767248fe1c87ec9a5f1809c079dd29ada" exitCode=0
Sep 18 19:10:54.612124 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:54.611800    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf" event={"ID":"3ea08ae6-8523-4108-b93e-e772d930bf75","Type":"ContainerDied","Data":"686b3b55e865eb78c50f60ebcfde254767248fe1c87ec9a5f1809c079dd29ada"}
Sep 18 19:10:54.612124 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:54.611850    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf" event={"ID":"3ea08ae6-8523-4108-b93e-e772d930bf75","Type":"ContainerStarted","Data":"823185baf39e01fbb3b994d2f4b4c5814893d42a1520f35606eb52ce30671bc4"}
Sep 18 19:10:55.757070 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:55.757042    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf"
Sep 18 19:10:55.872259 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:55.872131    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea08ae6-8523-4108-b93e-e772d930bf75-maas-api-service-ca\") pod \"3ea08ae6-8523-4108-b93e-e772d930bf75\" (UID: \"3ea08ae6-8523-4108-b93e-e772d930bf75\") "
Sep 18 19:10:55.872509 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:55.872487    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea08ae6-8523-4108-b93e-e772d930bf75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ea08ae6-8523-4108-b93e-e772d930bf75" (UID: "3ea08ae6-8523-4108-b93e-e772d930bf75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:10:55.973269 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:55.973229    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea08ae6-8523-4108-b93e-e772d930bf75-maas-api-service-ca\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:10:56.625506 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:56.625479    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf"
Sep 18 19:10:56.625674 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:56.625478    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf" event={"ID":"3ea08ae6-8523-4108-b93e-e772d930bf75","Type":"ContainerDied","Data":"823185baf39e01fbb3b994d2f4b4c5814893d42a1520f35606eb52ce30671bc4"}
Sep 18 19:10:56.625674 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:10:56.625610    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="823185baf39e01fbb3b994d2f4b4c5814893d42a1520f35606eb52ce30671bc4"
Sep 18 19:11:03.188504 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:03.188466    2558 scope.go:117] "RemoveContainer" containerID="c6c4b94f9ec708e2ec5501cde184e0cdbce469b59aaa162c7aea93bd15adb20e"
Sep 18 19:11:03.198830 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:03.198809    2558 scope.go:117] "RemoveContainer" containerID="31dca2a099c690f097eb193412e0ee22119aaaf46389b8e5edfcddf8b8f0f3d1"
Sep 18 19:11:03.776652 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:03.776615    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf"]
Sep 18 19:11:03.779180 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:03.779147    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a6d509-7g5wf"]
Sep 18 19:11:04.575578 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:04.575543    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ea08ae6-8523-4108-b93e-e772d930bf75" path="/var/lib/kubelet/pods/3ea08ae6-8523-4108-b93e-e772d930bf75/volumes"
Sep 18 19:11:10.449710 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:10.449672    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"]
Sep 18 19:11:10.450211 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:10.449928    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29" podUID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerName="istio-proxy" containerID="cri-o://0ca9ff965c88c1c96086aec731a04d53fb197f85f39c86a8b0e3ceaf93019f26" gracePeriod=30
Sep 18 19:11:13.320768 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:13.320728    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 19:11:13.321148 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:13.320792    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29" podUID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:11:15.705291 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.705264    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 19:11:15.749883 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.749796    2558 generic.go:356] "Generic (PLEG): container finished" podID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerID="0ca9ff965c88c1c96086aec731a04d53fb197f85f39c86a8b0e3ceaf93019f26" exitCode=0
Sep 18 19:11:15.749883 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.749862    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"
Sep 18 19:11:15.750075 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.749880    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29" event={"ID":"4de4ecae-a309-4b46-9562-2a532f0fb609","Type":"ContainerDied","Data":"0ca9ff965c88c1c96086aec731a04d53fb197f85f39c86a8b0e3ceaf93019f26"}
Sep 18 19:11:15.750075 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.749937    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29" event={"ID":"4de4ecae-a309-4b46-9562-2a532f0fb609","Type":"ContainerDied","Data":"55ff8f9d228a757fff4b2a16ae09386002893003b17d50b40f813adf51dcb283"}
Sep 18 19:11:15.750075 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.749965    2558 scope.go:117] "RemoveContainer" containerID="0ca9ff965c88c1c96086aec731a04d53fb197f85f39c86a8b0e3ceaf93019f26"
Sep 18 19:11:15.759791 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.759775    2558 scope.go:117] "RemoveContainer" containerID="44918cfabe3bddf252503e1e822e57a1d70a728fef00d1801683ba84893965ff"
Sep 18 19:11:15.768336 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.768317    2558 scope.go:117] "RemoveContainer" containerID="0ca9ff965c88c1c96086aec731a04d53fb197f85f39c86a8b0e3ceaf93019f26"
Sep 18 19:11:15.768589 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:11:15.768571    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca9ff965c88c1c96086aec731a04d53fb197f85f39c86a8b0e3ceaf93019f26\": container with ID starting with 0ca9ff965c88c1c96086aec731a04d53fb197f85f39c86a8b0e3ceaf93019f26 not found: ID does not exist" containerID="0ca9ff965c88c1c96086aec731a04d53fb197f85f39c86a8b0e3ceaf93019f26"
Sep 18 19:11:15.768638 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.768598    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca9ff965c88c1c96086aec731a04d53fb197f85f39c86a8b0e3ceaf93019f26"} err="failed to get container status \"0ca9ff965c88c1c96086aec731a04d53fb197f85f39c86a8b0e3ceaf93019f26\": rpc error: code = NotFound desc = could not find container \"0ca9ff965c88c1c96086aec731a04d53fb197f85f39c86a8b0e3ceaf93019f26\": container with ID starting with 0ca9ff965c88c1c96086aec731a04d53fb197f85f39c86a8b0e3ceaf93019f26 not found: ID does not exist"
Sep 18 19:11:15.768638 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.768617    2558 scope.go:117] "RemoveContainer" containerID="44918cfabe3bddf252503e1e822e57a1d70a728fef00d1801683ba84893965ff"
Sep 18 19:11:15.768866 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:11:15.768851    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44918cfabe3bddf252503e1e822e57a1d70a728fef00d1801683ba84893965ff\": container with ID starting with 44918cfabe3bddf252503e1e822e57a1d70a728fef00d1801683ba84893965ff not found: ID does not exist" containerID="44918cfabe3bddf252503e1e822e57a1d70a728fef00d1801683ba84893965ff"
Sep 18 19:11:15.768909 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.768870    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44918cfabe3bddf252503e1e822e57a1d70a728fef00d1801683ba84893965ff"} err="failed to get container status \"44918cfabe3bddf252503e1e822e57a1d70a728fef00d1801683ba84893965ff\": rpc error: code = NotFound desc = could not find container \"44918cfabe3bddf252503e1e822e57a1d70a728fef00d1801683ba84893965ff\": container with ID starting with 44918cfabe3bddf252503e1e822e57a1d70a728fef00d1801683ba84893965ff not found: ID does not exist"
Sep 18 19:11:15.861035 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861004    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-workload-certs\") pod \"4de4ecae-a309-4b46-9562-2a532f0fb609\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") "
Sep 18 19:11:15.861035 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861037    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de4ecae-a309-4b46-9562-2a532f0fb609-istiod-ca-cert\") pod \"4de4ecae-a309-4b46-9562-2a532f0fb609\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") "
Sep 18 19:11:15.861308 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861062    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-data\") pod \"4de4ecae-a309-4b46-9562-2a532f0fb609\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") "
Sep 18 19:11:15.861308 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861115    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-envoy\") pod \"4de4ecae-a309-4b46-9562-2a532f0fb609\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") "
Sep 18 19:11:15.861308 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861172    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-credential-socket\") pod \"4de4ecae-a309-4b46-9562-2a532f0fb609\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") "
Sep 18 19:11:15.861308 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861234    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-token\") pod \"4de4ecae-a309-4b46-9562-2a532f0fb609\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") "
Sep 18 19:11:15.861308 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861272    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-workload-socket\") pod \"4de4ecae-a309-4b46-9562-2a532f0fb609\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") "
Sep 18 19:11:15.861308 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861297    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47fgr\" (UniqueName: \"kubernetes.io/projected/4de4ecae-a309-4b46-9562-2a532f0fb609-kube-api-access-47fgr\") pod \"4de4ecae-a309-4b46-9562-2a532f0fb609\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") "
Sep 18 19:11:15.861680 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861351    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-podinfo\") pod \"4de4ecae-a309-4b46-9562-2a532f0fb609\" (UID: \"4de4ecae-a309-4b46-9562-2a532f0fb609\") "
Sep 18 19:11:15.861680 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861446    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-data" (OuterVolumeSpecName: "istio-data") pod "4de4ecae-a309-4b46-9562-2a532f0fb609" (UID: "4de4ecae-a309-4b46-9562-2a532f0fb609"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:11:15.861680 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861422    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4de4ecae-a309-4b46-9562-2a532f0fb609" (UID: "4de4ecae-a309-4b46-9562-2a532f0fb609"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:11:15.861680 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861467    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de4ecae-a309-4b46-9562-2a532f0fb609-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4de4ecae-a309-4b46-9562-2a532f0fb609" (UID: "4de4ecae-a309-4b46-9562-2a532f0fb609"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:11:15.861680 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861468    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4de4ecae-a309-4b46-9562-2a532f0fb609" (UID: "4de4ecae-a309-4b46-9562-2a532f0fb609"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:11:15.861680 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861643    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4de4ecae-a309-4b46-9562-2a532f0fb609" (UID: "4de4ecae-a309-4b46-9562-2a532f0fb609"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:11:15.862016 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861780    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-credential-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:11:15.862016 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861800    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-workload-socket\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:11:15.862016 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861813    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-workload-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:11:15.862016 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861826    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de4ecae-a309-4b46-9562-2a532f0fb609-istiod-ca-cert\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:11:15.862016 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.861837    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-data\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:11:15.863450 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.863426    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4de4ecae-a309-4b46-9562-2a532f0fb609" (UID: "4de4ecae-a309-4b46-9562-2a532f0fb609"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:11:15.863620 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.863603    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4de4ecae-a309-4b46-9562-2a532f0fb609" (UID: "4de4ecae-a309-4b46-9562-2a532f0fb609"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 19:11:15.863669 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.863618    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de4ecae-a309-4b46-9562-2a532f0fb609-kube-api-access-47fgr" (OuterVolumeSpecName: "kube-api-access-47fgr") pod "4de4ecae-a309-4b46-9562-2a532f0fb609" (UID: "4de4ecae-a309-4b46-9562-2a532f0fb609"). InnerVolumeSpecName "kube-api-access-47fgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:11:15.863669 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.863625    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-token" (OuterVolumeSpecName: "istio-token") pod "4de4ecae-a309-4b46-9562-2a532f0fb609" (UID: "4de4ecae-a309-4b46-9562-2a532f0fb609"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:11:15.962365 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.962332    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-envoy\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:11:15.962365 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.962359    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-token\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:11:15.962365 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.962368    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47fgr\" (UniqueName: \"kubernetes.io/projected/4de4ecae-a309-4b46-9562-2a532f0fb609-kube-api-access-47fgr\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:11:15.962589 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:15.962378    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de4ecae-a309-4b46-9562-2a532f0fb609-istio-podinfo\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:11:16.073797 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:16.073767    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"]
Sep 18 19:11:16.078970 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:16.078944    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a6d509-openshift-default-84969b5cf4-dhb29"]
Sep 18 19:11:16.575798 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:11:16.575734    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4de4ecae-a309-4b46-9562-2a532f0fb609" path="/var/lib/kubelet/pods/4de4ecae-a309-4b46-9562-2a532f0fb609/volumes"
Sep 18 19:12:03.236435 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:12:03.236324    2558 scope.go:117] "RemoveContainer" containerID="d361ca7feefc6aaf30aced2a7bc593588de7dea740380355f925851d6ef344d2"
Sep 18 19:12:03.245113 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:12:03.245095    2558 scope.go:117] "RemoveContainer" containerID="810a108eea442344feee7991b19cbbf26400ba8de0af90024caed98d81e47a2a"
Sep 18 19:12:03.253415 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:12:03.253395    2558 scope.go:117] "RemoveContainer" containerID="6e6bc8f92a39f6a8ac9982bab381ae71305ba50b94fae52a7110df70a050949a"
Sep 18 19:13:41.273503 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:41.273466    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"]
Sep 18 19:13:41.273957 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:41.273676    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p" podUID="792ae493-6c54-472b-bd71-f144fd3f28e3" containerName="manager" containerID="cri-o://5c7682f89f129ec04bbad57ebbb4c577a25bf493d74eedcb48ae1db9a888d794" gracePeriod=10
Sep 18 19:13:43.617746 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.617717    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"
Sep 18 19:13:43.620303 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.620278    2558 generic.go:356] "Generic (PLEG): container finished" podID="792ae493-6c54-472b-bd71-f144fd3f28e3" containerID="5c7682f89f129ec04bbad57ebbb4c577a25bf493d74eedcb48ae1db9a888d794" exitCode=0
Sep 18 19:13:43.620430 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.620325    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"
Sep 18 19:13:43.620430 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.620366    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p" event={"ID":"792ae493-6c54-472b-bd71-f144fd3f28e3","Type":"ContainerDied","Data":"5c7682f89f129ec04bbad57ebbb4c577a25bf493d74eedcb48ae1db9a888d794"}
Sep 18 19:13:43.620566 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.620431    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p" event={"ID":"792ae493-6c54-472b-bd71-f144fd3f28e3","Type":"ContainerDied","Data":"b46c7e73760f0654c68af85dd6aa3a6c6f41846f75a2eb2245980ed905875b8e"}
Sep 18 19:13:43.620566 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.620464    2558 scope.go:117] "RemoveContainer" containerID="5c7682f89f129ec04bbad57ebbb4c577a25bf493d74eedcb48ae1db9a888d794"
Sep 18 19:13:43.630223 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.630201    2558 scope.go:117] "RemoveContainer" containerID="5c7682f89f129ec04bbad57ebbb4c577a25bf493d74eedcb48ae1db9a888d794"
Sep 18 19:13:43.630512 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:13:43.630495    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c7682f89f129ec04bbad57ebbb4c577a25bf493d74eedcb48ae1db9a888d794\": container with ID starting with 5c7682f89f129ec04bbad57ebbb4c577a25bf493d74eedcb48ae1db9a888d794 not found: ID does not exist" containerID="5c7682f89f129ec04bbad57ebbb4c577a25bf493d74eedcb48ae1db9a888d794"
Sep 18 19:13:43.630567 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.630520    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c7682f89f129ec04bbad57ebbb4c577a25bf493d74eedcb48ae1db9a888d794"} err="failed to get container status \"5c7682f89f129ec04bbad57ebbb4c577a25bf493d74eedcb48ae1db9a888d794\": rpc error: code = NotFound desc = could not find container \"5c7682f89f129ec04bbad57ebbb4c577a25bf493d74eedcb48ae1db9a888d794\": container with ID starting with 5c7682f89f129ec04bbad57ebbb4c577a25bf493d74eedcb48ae1db9a888d794 not found: ID does not exist"
Sep 18 19:13:43.772465 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.772376    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/792ae493-6c54-472b-bd71-f144fd3f28e3-extensions-socket-volume\") pod \"792ae493-6c54-472b-bd71-f144fd3f28e3\" (UID: \"792ae493-6c54-472b-bd71-f144fd3f28e3\") "
Sep 18 19:13:43.772633 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.772512    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqq5b\" (UniqueName: \"kubernetes.io/projected/792ae493-6c54-472b-bd71-f144fd3f28e3-kube-api-access-jqq5b\") pod \"792ae493-6c54-472b-bd71-f144fd3f28e3\" (UID: \"792ae493-6c54-472b-bd71-f144fd3f28e3\") "
Sep 18 19:13:43.772833 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.772810    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792ae493-6c54-472b-bd71-f144fd3f28e3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "792ae493-6c54-472b-bd71-f144fd3f28e3" (UID: "792ae493-6c54-472b-bd71-f144fd3f28e3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:13:43.774602 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.774580    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792ae493-6c54-472b-bd71-f144fd3f28e3-kube-api-access-jqq5b" (OuterVolumeSpecName: "kube-api-access-jqq5b") pod "792ae493-6c54-472b-bd71-f144fd3f28e3" (UID: "792ae493-6c54-472b-bd71-f144fd3f28e3"). InnerVolumeSpecName "kube-api-access-jqq5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:13:43.874280 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.874240    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/792ae493-6c54-472b-bd71-f144fd3f28e3-extensions-socket-volume\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:13:43.874280 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.874273    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqq5b\" (UniqueName: \"kubernetes.io/projected/792ae493-6c54-472b-bd71-f144fd3f28e3-kube-api-access-jqq5b\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:13:43.945694 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.945665    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"]
Sep 18 19:13:43.953227 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:43.953202    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tzr6p"]
Sep 18 19:13:44.575124 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:13:44.575092    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="792ae493-6c54-472b-bd71-f144fd3f28e3" path="/var/lib/kubelet/pods/792ae493-6c54-472b-bd71-f144fd3f28e3/volumes"
Sep 18 19:14:02.802652 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:02.802622    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 19:14:02.805298 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:02.805277    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 19:14:02.806208 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:02.806173    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 19:14:02.809041 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:02.809025    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l6vdn_9693f61e-8c2e-4af7-8ff4-fbe6ef70f7bd/ovn-acl-logging/0.log"
Sep 18 19:14:41.977029 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.976994    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp"]
Sep 18 19:14:41.977494 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.977472    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea08ae6-8523-4108-b93e-e772d930bf75" containerName="revoke-keys"
Sep 18 19:14:41.977494 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.977486    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea08ae6-8523-4108-b93e-e772d930bf75" containerName="revoke-keys"
Sep 18 19:14:41.977494 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.977495    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerName="istio-proxy"
Sep 18 19:14:41.977600 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.977501    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerName="istio-proxy"
Sep 18 19:14:41.977600 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.977508    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerName="istio-proxy"
Sep 18 19:14:41.977600 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.977513    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerName="istio-proxy"
Sep 18 19:14:41.977600 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.977525    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792ae493-6c54-472b-bd71-f144fd3f28e3" containerName="manager"
Sep 18 19:14:41.977600 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.977533    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="792ae493-6c54-472b-bd71-f144fd3f28e3" containerName="manager"
Sep 18 19:14:41.977752 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.977606    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerName="istio-proxy"
Sep 18 19:14:41.977752 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.977621    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ea08ae6-8523-4108-b93e-e772d930bf75" containerName="revoke-keys"
Sep 18 19:14:41.977752 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.977628    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="792ae493-6c54-472b-bd71-f144fd3f28e3" containerName="manager"
Sep 18 19:14:41.980861 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.980841    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp"
Sep 18 19:14:41.983492 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.983470    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-99vf4\""
Sep 18 19:14:41.992629 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.992601    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4m2m\" (UniqueName: \"kubernetes.io/projected/a16bad66-b9aa-48a9-b3ff-c8935790d72c-kube-api-access-k4m2m\") pod \"kuadrant-operator-controller-manager-5769f99d9f-swmnp\" (UID: \"a16bad66-b9aa-48a9-b3ff-c8935790d72c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp"
Sep 18 19:14:41.992886 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.992865    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a16bad66-b9aa-48a9-b3ff-c8935790d72c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-swmnp\" (UID: \"a16bad66-b9aa-48a9-b3ff-c8935790d72c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp"
Sep 18 19:14:41.997274 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:41.997248    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp"]
Sep 18 19:14:42.093742 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:42.093690    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4m2m\" (UniqueName: \"kubernetes.io/projected/a16bad66-b9aa-48a9-b3ff-c8935790d72c-kube-api-access-k4m2m\") pod \"kuadrant-operator-controller-manager-5769f99d9f-swmnp\" (UID: \"a16bad66-b9aa-48a9-b3ff-c8935790d72c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp"
Sep 18 19:14:42.093921 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:42.093769    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a16bad66-b9aa-48a9-b3ff-c8935790d72c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-swmnp\" (UID: \"a16bad66-b9aa-48a9-b3ff-c8935790d72c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp"
Sep 18 19:14:42.094133 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:42.094115    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a16bad66-b9aa-48a9-b3ff-c8935790d72c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-swmnp\" (UID: \"a16bad66-b9aa-48a9-b3ff-c8935790d72c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp"
Sep 18 19:14:42.102330 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:42.102304    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4m2m\" (UniqueName: \"kubernetes.io/projected/a16bad66-b9aa-48a9-b3ff-c8935790d72c-kube-api-access-k4m2m\") pod \"kuadrant-operator-controller-manager-5769f99d9f-swmnp\" (UID: \"a16bad66-b9aa-48a9-b3ff-c8935790d72c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp"
Sep 18 19:14:42.298560 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:42.298462    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp"
Sep 18 19:14:42.585946 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:42.585919    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp"]
Sep 18 19:14:42.586992 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:14:42.586966    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16bad66_b9aa_48a9_b3ff_c8935790d72c.slice/crio-feeb516c451a5d7f1fc28b87824d154d3c091eb83bc4bebf88b91d37f532c4c1 WatchSource:0}: Error finding container feeb516c451a5d7f1fc28b87824d154d3c091eb83bc4bebf88b91d37f532c4c1: Status 404 returned error can't find the container with id feeb516c451a5d7f1fc28b87824d154d3c091eb83bc4bebf88b91d37f532c4c1
Sep 18 19:14:42.969335 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:42.969299    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp" event={"ID":"a16bad66-b9aa-48a9-b3ff-c8935790d72c","Type":"ContainerStarted","Data":"9ae8299b119b5ca0d05a139a9fbb96d9094c88d3667aaa03b7fb0210625a60a4"}
Sep 18 19:14:42.969543 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:42.969411    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp"
Sep 18 19:14:42.969543 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:42.969432    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp" event={"ID":"a16bad66-b9aa-48a9-b3ff-c8935790d72c","Type":"ContainerStarted","Data":"feeb516c451a5d7f1fc28b87824d154d3c091eb83bc4bebf88b91d37f532c4c1"}
Sep 18 19:14:42.991381 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:42.991328    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp" podStartSLOduration=1.9913104929999998 podStartE2EDuration="1.991310493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:42.989315181 +0000 UTC m=+2142.425894024" watchObservedRunningTime="2026-09-18 19:14:42.991310493 +0000 UTC m=+2142.427889378"
Sep 18 19:14:53.978123 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:14:53.978036    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-swmnp"
Sep 18 19:15:03.306545 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:03.306513    2558 scope.go:117] "RemoveContainer" containerID="c60fa1430d14a9fa96021779074e1c465d71849a60d42043849919ebbd4de608"
Sep 18 19:15:20.458986 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.458950    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"]
Sep 18 19:15:20.459487 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.459436    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4de4ecae-a309-4b46-9562-2a532f0fb609" containerName="istio-proxy"
Sep 18 19:15:20.464122 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.464100    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.466658 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.466638    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e93e024f-kserve-self-signed-certs\""
Sep 18 19:15:20.472076 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.472055    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"]
Sep 18 19:15:20.650302 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.650265    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-dshm\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.650494 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.650324    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-model-cache\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.650494 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.650407    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r7rg\" (UniqueName: \"kubernetes.io/projected/40925aba-cc90-480b-85ab-2c83b161257e-kube-api-access-6r7rg\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.650494 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.650482    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-home\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.650619 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.650540    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-tmp-dir\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.650684 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.650664    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40925aba-cc90-480b-85ab-2c83b161257e-tls-certs\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.751738 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.751651    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-model-cache\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.751738 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.751698    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7rg\" (UniqueName: \"kubernetes.io/projected/40925aba-cc90-480b-85ab-2c83b161257e-kube-api-access-6r7rg\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.751967 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.751756    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-home\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.751967 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.751792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-tmp-dir\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.751967 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.751911    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40925aba-cc90-480b-85ab-2c83b161257e-tls-certs\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.752119 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.751986    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-dshm\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.752175 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.752150    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-home\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.752260 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.752228    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-tmp-dir\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.752379 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.752356    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-model-cache\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.754458 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.754430    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-dshm\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.754688 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.754672    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40925aba-cc90-480b-85ab-2c83b161257e-tls-certs\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.761948 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.761926    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r7rg\" (UniqueName: \"kubernetes.io/projected/40925aba-cc90-480b-85ab-2c83b161257e-kube-api-access-6r7rg\") pod \"e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") " pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:20.774785 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:20.774765    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:21.053694 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:21.053665    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"]
Sep 18 19:15:21.055572 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:15:21.055541    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40925aba_cc90_480b_85ab_2c83b161257e.slice/crio-87b139a226dbda1e5cebe174badff218d74903bce355218ef98a2318b6405288 WatchSource:0}: Error finding container 87b139a226dbda1e5cebe174badff218d74903bce355218ef98a2318b6405288: Status 404 returned error can't find the container with id 87b139a226dbda1e5cebe174badff218d74903bce355218ef98a2318b6405288
Sep 18 19:15:21.201803 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:21.201767    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29" event={"ID":"40925aba-cc90-480b-85ab-2c83b161257e","Type":"ContainerStarted","Data":"96911ee57825b31816c4afd41452430cfd557cbdbdc9e7582fb499b5c88c39ae"}
Sep 18 19:15:21.201985 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:21.201813    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29" event={"ID":"40925aba-cc90-480b-85ab-2c83b161257e","Type":"ContainerStarted","Data":"87b139a226dbda1e5cebe174badff218d74903bce355218ef98a2318b6405288"}
Sep 18 19:15:21.221456 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:21.221408    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29" podStartSLOduration=1.221391577 podStartE2EDuration="1.221391577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:21.219280346 +0000 UTC m=+2180.655859188" watchObservedRunningTime="2026-09-18 19:15:21.221391577 +0000 UTC m=+2180.657970418"
Sep 18 19:15:30.775731 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:30.775693    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:30.775731 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:30.775738    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:30.788794 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:30.788763    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:31.272639 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:31.272609    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:15:53.738348 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:15:53.738313    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"]
Sep 18 19:16:03.329395 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:03.329363    2558 scope.go:117] "RemoveContainer" containerID="6a4207c65645ac312ac76ec767d2d6efbec8a3c6a63214b98ee7c2bda77856a9"
Sep 18 19:16:08.758759 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:08.758714    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29" podUID="40925aba-cc90-480b-85ab-2c83b161257e" containerName="main" containerID="cri-o://96911ee57825b31816c4afd41452430cfd557cbdbdc9e7582fb499b5c88c39ae" gracePeriod=45
Sep 18 19:16:09.001931 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.001907    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:16:09.090406 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.090379    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-model-cache\") pod \"40925aba-cc90-480b-85ab-2c83b161257e\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") "
Sep 18 19:16:09.090558 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.090449    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-dshm\") pod \"40925aba-cc90-480b-85ab-2c83b161257e\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") "
Sep 18 19:16:09.090558 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.090466    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-home\") pod \"40925aba-cc90-480b-85ab-2c83b161257e\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") "
Sep 18 19:16:09.090558 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.090512    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-tmp-dir\") pod \"40925aba-cc90-480b-85ab-2c83b161257e\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") "
Sep 18 19:16:09.090695 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.090565    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r7rg\" (UniqueName: \"kubernetes.io/projected/40925aba-cc90-480b-85ab-2c83b161257e-kube-api-access-6r7rg\") pod \"40925aba-cc90-480b-85ab-2c83b161257e\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") "
Sep 18 19:16:09.090695 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.090583    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40925aba-cc90-480b-85ab-2c83b161257e-tls-certs\") pod \"40925aba-cc90-480b-85ab-2c83b161257e\" (UID: \"40925aba-cc90-480b-85ab-2c83b161257e\") "
Sep 18 19:16:09.090695 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.090681    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-model-cache" (OuterVolumeSpecName: "model-cache") pod "40925aba-cc90-480b-85ab-2c83b161257e" (UID: "40925aba-cc90-480b-85ab-2c83b161257e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:16:09.090852 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.090723    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-home" (OuterVolumeSpecName: "home") pod "40925aba-cc90-480b-85ab-2c83b161257e" (UID: "40925aba-cc90-480b-85ab-2c83b161257e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:16:09.090852 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.090827    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40925aba-cc90-480b-85ab-2c83b161257e" (UID: "40925aba-cc90-480b-85ab-2c83b161257e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:16:09.090964 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.090871    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-home\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:16:09.090964 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.090884    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-model-cache\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:16:09.092655 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.092630    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-dshm" (OuterVolumeSpecName: "dshm") pod "40925aba-cc90-480b-85ab-2c83b161257e" (UID: "40925aba-cc90-480b-85ab-2c83b161257e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:16:09.092779 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.092764    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40925aba-cc90-480b-85ab-2c83b161257e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40925aba-cc90-480b-85ab-2c83b161257e" (UID: "40925aba-cc90-480b-85ab-2c83b161257e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:16:09.092844 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.092789    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40925aba-cc90-480b-85ab-2c83b161257e-kube-api-access-6r7rg" (OuterVolumeSpecName: "kube-api-access-6r7rg") pod "40925aba-cc90-480b-85ab-2c83b161257e" (UID: "40925aba-cc90-480b-85ab-2c83b161257e"). InnerVolumeSpecName "kube-api-access-6r7rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:16:09.192041 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.191998    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r7rg\" (UniqueName: \"kubernetes.io/projected/40925aba-cc90-480b-85ab-2c83b161257e-kube-api-access-6r7rg\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:16:09.192041 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.192034    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40925aba-cc90-480b-85ab-2c83b161257e-tls-certs\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:16:09.192041 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.192048    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-dshm\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:16:09.192313 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.192059    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40925aba-cc90-480b-85ab-2c83b161257e-tmp-dir\") on node \"ip-10-0-130-78.ec2.internal\" DevicePath \"\""
Sep 18 19:16:09.492747 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.492653    2558 generic.go:356] "Generic (PLEG): container finished" podID="40925aba-cc90-480b-85ab-2c83b161257e" containerID="96911ee57825b31816c4afd41452430cfd557cbdbdc9e7582fb499b5c88c39ae" exitCode=0
Sep 18 19:16:09.492747 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.492706    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"
Sep 18 19:16:09.492936 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.492737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29" event={"ID":"40925aba-cc90-480b-85ab-2c83b161257e","Type":"ContainerDied","Data":"96911ee57825b31816c4afd41452430cfd557cbdbdc9e7582fb499b5c88c39ae"}
Sep 18 19:16:09.492936 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.492783    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29" event={"ID":"40925aba-cc90-480b-85ab-2c83b161257e","Type":"ContainerDied","Data":"87b139a226dbda1e5cebe174badff218d74903bce355218ef98a2318b6405288"}
Sep 18 19:16:09.492936 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.492804    2558 scope.go:117] "RemoveContainer" containerID="96911ee57825b31816c4afd41452430cfd557cbdbdc9e7582fb499b5c88c39ae"
Sep 18 19:16:09.502926 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.502904    2558 scope.go:117] "RemoveContainer" containerID="96911ee57825b31816c4afd41452430cfd557cbdbdc9e7582fb499b5c88c39ae"
Sep 18 19:16:09.503271 ip-10-0-130-78 kubenswrapper[2558]: E0918 19:16:09.503244    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96911ee57825b31816c4afd41452430cfd557cbdbdc9e7582fb499b5c88c39ae\": container with ID starting with 96911ee57825b31816c4afd41452430cfd557cbdbdc9e7582fb499b5c88c39ae not found: ID does not exist" containerID="96911ee57825b31816c4afd41452430cfd557cbdbdc9e7582fb499b5c88c39ae"
Sep 18 19:16:09.503389 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.503292    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96911ee57825b31816c4afd41452430cfd557cbdbdc9e7582fb499b5c88c39ae"} err="failed to get container status \"96911ee57825b31816c4afd41452430cfd557cbdbdc9e7582fb499b5c88c39ae\": rpc error: code = NotFound desc = could not find container \"96911ee57825b31816c4afd41452430cfd557cbdbdc9e7582fb499b5c88c39ae\": container with ID starting with 96911ee57825b31816c4afd41452430cfd557cbdbdc9e7582fb499b5c88c39ae not found: ID does not exist"
Sep 18 19:16:09.515010 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.514982    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"]
Sep 18 19:16:09.518583 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:09.518563    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e93e024f-kserve-556578f5dc-xvq29"]
Sep 18 19:16:10.576177 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:10.576143    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40925aba-cc90-480b-85ab-2c83b161257e" path="/var/lib/kubelet/pods/40925aba-cc90-480b-85ab-2c83b161257e/volumes"
Sep 18 19:16:41.581766 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.581731    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g6kgh/must-gather-kk6gr"]
Sep 18 19:16:41.582171 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.582102    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40925aba-cc90-480b-85ab-2c83b161257e" containerName="main"
Sep 18 19:16:41.582171 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.582114    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="40925aba-cc90-480b-85ab-2c83b161257e" containerName="main"
Sep 18 19:16:41.582269 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.582202    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="40925aba-cc90-480b-85ab-2c83b161257e" containerName="main"
Sep 18 19:16:41.585466 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.585450    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g6kgh/must-gather-kk6gr"
Sep 18 19:16:41.587801 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.587780    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g6kgh\"/\"default-dockercfg-xkzgg\""
Sep 18 19:16:41.587896 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.587780    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g6kgh\"/\"openshift-service-ca.crt\""
Sep 18 19:16:41.588061 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.588047    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g6kgh\"/\"kube-root-ca.crt\""
Sep 18 19:16:41.595800 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.595777    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g6kgh/must-gather-kk6gr"]
Sep 18 19:16:41.681722 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.681688    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2cb49271-1b26-4a7a-a23e-13ca2c618de0-must-gather-output\") pod \"must-gather-kk6gr\" (UID: \"2cb49271-1b26-4a7a-a23e-13ca2c618de0\") " pod="openshift-must-gather-g6kgh/must-gather-kk6gr"
Sep 18 19:16:41.681916 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.681764    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mj55\" (UniqueName: \"kubernetes.io/projected/2cb49271-1b26-4a7a-a23e-13ca2c618de0-kube-api-access-9mj55\") pod \"must-gather-kk6gr\" (UID: \"2cb49271-1b26-4a7a-a23e-13ca2c618de0\") " pod="openshift-must-gather-g6kgh/must-gather-kk6gr"
Sep 18 19:16:41.782425 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.782391    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2cb49271-1b26-4a7a-a23e-13ca2c618de0-must-gather-output\") pod \"must-gather-kk6gr\" (UID: \"2cb49271-1b26-4a7a-a23e-13ca2c618de0\") " pod="openshift-must-gather-g6kgh/must-gather-kk6gr"
Sep 18 19:16:41.782612 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.782466    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mj55\" (UniqueName: \"kubernetes.io/projected/2cb49271-1b26-4a7a-a23e-13ca2c618de0-kube-api-access-9mj55\") pod \"must-gather-kk6gr\" (UID: \"2cb49271-1b26-4a7a-a23e-13ca2c618de0\") " pod="openshift-must-gather-g6kgh/must-gather-kk6gr"
Sep 18 19:16:41.782755 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.782738    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2cb49271-1b26-4a7a-a23e-13ca2c618de0-must-gather-output\") pod \"must-gather-kk6gr\" (UID: \"2cb49271-1b26-4a7a-a23e-13ca2c618de0\") " pod="openshift-must-gather-g6kgh/must-gather-kk6gr"
Sep 18 19:16:41.791052 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.791027    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mj55\" (UniqueName: \"kubernetes.io/projected/2cb49271-1b26-4a7a-a23e-13ca2c618de0-kube-api-access-9mj55\") pod \"must-gather-kk6gr\" (UID: \"2cb49271-1b26-4a7a-a23e-13ca2c618de0\") " pod="openshift-must-gather-g6kgh/must-gather-kk6gr"
Sep 18 19:16:41.896071 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:41.896037    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g6kgh/must-gather-kk6gr"
Sep 18 19:16:42.171962 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:42.171935    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g6kgh/must-gather-kk6gr"]
Sep 18 19:16:42.173657 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:16:42.173618    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cb49271_1b26_4a7a_a23e_13ca2c618de0.slice/crio-9443d4d6f614f2bba900d6100b9dbb6120bd24c0b209eee5fbfb7cb73be23a39 WatchSource:0}: Error finding container 9443d4d6f614f2bba900d6100b9dbb6120bd24c0b209eee5fbfb7cb73be23a39: Status 404 returned error can't find the container with id 9443d4d6f614f2bba900d6100b9dbb6120bd24c0b209eee5fbfb7cb73be23a39
Sep 18 19:16:42.175831 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:42.175811    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:16:42.690651 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:42.690613    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g6kgh/must-gather-kk6gr" event={"ID":"2cb49271-1b26-4a7a-a23e-13ca2c618de0","Type":"ContainerStarted","Data":"9443d4d6f614f2bba900d6100b9dbb6120bd24c0b209eee5fbfb7cb73be23a39"}
Sep 18 19:16:43.702845 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:43.702805    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g6kgh/must-gather-kk6gr" event={"ID":"2cb49271-1b26-4a7a-a23e-13ca2c618de0","Type":"ContainerStarted","Data":"034f4e1d08c05dd237944b0e217695c7e4fd670f4977f1d34b2c408658777476"}
Sep 18 19:16:43.703412 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:43.703383    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g6kgh/must-gather-kk6gr" event={"ID":"2cb49271-1b26-4a7a-a23e-13ca2c618de0","Type":"ContainerStarted","Data":"c087390d8c3b5b5784596a3ee2ff8e08fe793cdebb9bee55105a7c72309b7e46"}
Sep 18 19:16:43.722390 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:43.722338    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g6kgh/must-gather-kk6gr" podStartSLOduration=1.8889731410000001 podStartE2EDuration="2.722321899s" totalImagesPullingTime="833.348758ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:43.719476711 +0000 UTC m=+2263.156055565" watchObservedRunningTime="2026-09-18 19:16:43.722321899 +0000 UTC m=+2263.158900743"
Sep 18 19:16:44.737598 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:44.737570    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w9bz8_9ac6087e-296b-4917-bee6-8540193b8719/global-pull-secret-syncer/0.log"
Sep 18 19:16:44.872734 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:44.872693    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xmrrh_077f92f7-ee39-40c4-9873-43b00c9141a9/konnectivity-agent/0.log"
Sep 18 19:16:44.925906 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:44.925878    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-78.ec2.internal_8aa0497c56ca99c73b6260acf272ac5b/haproxy/0.log"
Sep 18 19:16:49.190266 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:49.190227    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-27v92_cf4cff2f-a2c3-4e90-b08c-3100811d01be/manager/0.log"
Sep 18 19:16:49.262399 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:49.262358    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-swmnp_a16bad66-b9aa-48a9-b3ff-c8935790d72c/manager/0.log"
Sep 18 19:16:49.416915 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:49.416879    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-mcjlp_ae6bba30-f072-4a01-83c0-80dacef1565b/manager/0.log"
Sep 18 19:16:51.320728 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:51.320692    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-p27wv_9b2a9f30-7931-4049-8d50-64203df5c3fd/cluster-monitoring-operator/0.log"
Sep 18 19:16:52.049171 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:52.049085    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ttrkh_1b347122-1593-4a1a-b376-e0423c45ac44/node-exporter/0.log"
Sep 18 19:16:52.082491 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:52.082445    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ttrkh_1b347122-1593-4a1a-b376-e0423c45ac44/kube-rbac-proxy/0.log"
Sep 18 19:16:52.155024 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:52.154993    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ttrkh_1b347122-1593-4a1a-b376-e0423c45ac44/init-textfile/0.log"
Sep 18 19:16:52.801920 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:52.801890    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d44ccc684-h79gv_f2b79b04-761c-4dc3-829e-a3a779aac406/telemeter-client/0.log"
Sep 18 19:16:52.824749 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:52.824713    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d44ccc684-h79gv_f2b79b04-761c-4dc3-829e-a3a779aac406/reload/0.log"
Sep 18 19:16:52.846625 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:52.846578    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d44ccc684-h79gv_f2b79b04-761c-4dc3-829e-a3a779aac406/kube-rbac-proxy/0.log"
Sep 18 19:16:54.305608 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.305577    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-4lcv7_f649578f-b8d4-4207-9dcc-69ce583fc6f6/networking-console-plugin/0.log"
Sep 18 19:16:54.498678 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.498640    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"]
Sep 18 19:16:54.504322 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.504292    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.548337 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.548295    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"]
Sep 18 19:16:54.619858 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.619811    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd204067-03a8-4785-9c9c-98a0a6d2f0e8-sys\") pod \"perf-node-gather-daemonset-4mwxp\" (UID: \"cd204067-03a8-4785-9c9c-98a0a6d2f0e8\") " pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.620072 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.619981    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cd204067-03a8-4785-9c9c-98a0a6d2f0e8-proc\") pod \"perf-node-gather-daemonset-4mwxp\" (UID: \"cd204067-03a8-4785-9c9c-98a0a6d2f0e8\") " pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.620072 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.620042    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd204067-03a8-4785-9c9c-98a0a6d2f0e8-lib-modules\") pod \"perf-node-gather-daemonset-4mwxp\" (UID: \"cd204067-03a8-4785-9c9c-98a0a6d2f0e8\") " pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.620072 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.620067    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h66wd\" (UniqueName: \"kubernetes.io/projected/cd204067-03a8-4785-9c9c-98a0a6d2f0e8-kube-api-access-h66wd\") pod \"perf-node-gather-daemonset-4mwxp\" (UID: \"cd204067-03a8-4785-9c9c-98a0a6d2f0e8\") " pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.620248 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.620094    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cd204067-03a8-4785-9c9c-98a0a6d2f0e8-podres\") pod \"perf-node-gather-daemonset-4mwxp\" (UID: \"cd204067-03a8-4785-9c9c-98a0a6d2f0e8\") " pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.721267 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.721226    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd204067-03a8-4785-9c9c-98a0a6d2f0e8-sys\") pod \"perf-node-gather-daemonset-4mwxp\" (UID: \"cd204067-03a8-4785-9c9c-98a0a6d2f0e8\") " pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.721473 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.721352    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cd204067-03a8-4785-9c9c-98a0a6d2f0e8-proc\") pod \"perf-node-gather-daemonset-4mwxp\" (UID: \"cd204067-03a8-4785-9c9c-98a0a6d2f0e8\") " pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.721473 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.721358    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd204067-03a8-4785-9c9c-98a0a6d2f0e8-sys\") pod \"perf-node-gather-daemonset-4mwxp\" (UID: \"cd204067-03a8-4785-9c9c-98a0a6d2f0e8\") " pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.721473 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.721421    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cd204067-03a8-4785-9c9c-98a0a6d2f0e8-proc\") pod \"perf-node-gather-daemonset-4mwxp\" (UID: \"cd204067-03a8-4785-9c9c-98a0a6d2f0e8\") " pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.721473 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.721425    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd204067-03a8-4785-9c9c-98a0a6d2f0e8-lib-modules\") pod \"perf-node-gather-daemonset-4mwxp\" (UID: \"cd204067-03a8-4785-9c9c-98a0a6d2f0e8\") " pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.721473 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.721453    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h66wd\" (UniqueName: \"kubernetes.io/projected/cd204067-03a8-4785-9c9c-98a0a6d2f0e8-kube-api-access-h66wd\") pod \"perf-node-gather-daemonset-4mwxp\" (UID: \"cd204067-03a8-4785-9c9c-98a0a6d2f0e8\") " pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.721786 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.721487    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cd204067-03a8-4785-9c9c-98a0a6d2f0e8-podres\") pod \"perf-node-gather-daemonset-4mwxp\" (UID: \"cd204067-03a8-4785-9c9c-98a0a6d2f0e8\") " pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.721786 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.721588    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd204067-03a8-4785-9c9c-98a0a6d2f0e8-lib-modules\") pod \"perf-node-gather-daemonset-4mwxp\" (UID: \"cd204067-03a8-4785-9c9c-98a0a6d2f0e8\") " pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.721786 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.721611    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cd204067-03a8-4785-9c9c-98a0a6d2f0e8-podres\") pod \"perf-node-gather-daemonset-4mwxp\" (UID: \"cd204067-03a8-4785-9c9c-98a0a6d2f0e8\") " pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.731167 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.731129    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h66wd\" (UniqueName: \"kubernetes.io/projected/cd204067-03a8-4785-9c9c-98a0a6d2f0e8-kube-api-access-h66wd\") pod \"perf-node-gather-daemonset-4mwxp\" (UID: \"cd204067-03a8-4785-9c9c-98a0a6d2f0e8\") " pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:54.817934 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:54.817898    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:55.057337 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:55.057302    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/1.log"
Sep 18 19:16:55.063120 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:55.063093    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-whw2g_8c3a44de-98c4-42e8-9ed2-7587293418ca/console-operator/2.log"
Sep 18 19:16:55.135176 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:55.135137    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"]
Sep 18 19:16:55.135914 ip-10-0-130-78 kubenswrapper[2558]: W0918 19:16:55.135884    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcd204067_03a8_4785_9c9c_98a0a6d2f0e8.slice/crio-997fc6b78d1af8ab49c4779d4d15a28c588a9dccd8c6036f32e33a269bd7bca0 WatchSource:0}: Error finding container 997fc6b78d1af8ab49c4779d4d15a28c588a9dccd8c6036f32e33a269bd7bca0: Status 404 returned error can't find the container with id 997fc6b78d1af8ab49c4779d4d15a28c588a9dccd8c6036f32e33a269bd7bca0
Sep 18 19:16:55.731400 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:55.731348    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-rk7wk_0801c387-848b-4c2f-8188-97b7720c46d8/download-server/0.log"
Sep 18 19:16:55.806640 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:55.806594    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp" event={"ID":"cd204067-03a8-4785-9c9c-98a0a6d2f0e8","Type":"ContainerStarted","Data":"10c8a8934dc7f0603c37964122ed5bd81cb8c807a57f846cd0886e10af87d8e5"}
Sep 18 19:16:55.806843 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:55.806662    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp" event={"ID":"cd204067-03a8-4785-9c9c-98a0a6d2f0e8","Type":"ContainerStarted","Data":"997fc6b78d1af8ab49c4779d4d15a28c588a9dccd8c6036f32e33a269bd7bca0"}
Sep 18 19:16:55.806943 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:55.806922    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:16:55.827342 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:55.827284    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp" podStartSLOduration=1.8272639069999999 podStartE2EDuration="1.827263907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:55.823275937 +0000 UTC m=+2275.259854779" watchObservedRunningTime="2026-09-18 19:16:55.827263907 +0000 UTC m=+2275.263842750"
Sep 18 19:16:56.427561 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:56.427525    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-cltgd_f0bb329d-2119-4411-bdbd-8c01fdcf493e/volume-data-source-validator/0.log"
Sep 18 19:16:57.998572 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:57.998538    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zw2cl_6455458b-83a0-4f35-857a-f11c3866299e/dns/0.log"
Sep 18 19:16:58.061262 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:58.061227    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zw2cl_6455458b-83a0-4f35-857a-f11c3866299e/kube-rbac-proxy/0.log"
Sep 18 19:16:58.225599 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:58.225570    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xbz24_aa59b9bc-694f-4b16-99bf-5d31a85bca74/dns-node-resolver/0.log"
Sep 18 19:16:58.954710 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:16:58.954680    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zmgsq_aad288b6-deba-464d-917f-dcc47d794865/node-ca/0.log"
Sep 18 19:17:00.093577 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:00.093546    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw_98840362-73e4-4d07-9b00-e2f02c2cb44f/istio-proxy/0.log"
Sep 18 19:17:00.929828 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:00.929797    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-fvbcs_a0b22143-f65e-4cf9-8674-14fe8946d89f/payload-pre-processing/0.log"
Sep 18 19:17:00.964308 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:00.964278    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-bbxqv_ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1/payload-processing/0.log"
Sep 18 19:17:00.990663 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:00.990631    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bd44bd444-kdgsw_7117c313-6fb4-48b0-8f3e-16e2c9c8d551/router/0.log"
Sep 18 19:17:01.828906 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:01.828876    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g6kgh/perf-node-gather-daemonset-4mwxp"
Sep 18 19:17:01.937075 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:01.937044    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-42z88_bf18e0b9-6a23-4338-893f-e8d7ca463c1e/serve-healthcheck-canary/0.log"
Sep 18 19:17:02.526417 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:02.526384    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-q64fh_63302165-e325-4efd-88d8-dda368f6066e/insights-operator/0.log"
Sep 18 19:17:02.527012 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:02.526991    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-q64fh_63302165-e325-4efd-88d8-dda368f6066e/insights-operator/1.log"
Sep 18 19:17:02.697804 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:02.697752    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wpbnx_629ab9c3-0c87-4f52-9770-c861b9ea5705/kube-rbac-proxy/0.log"
Sep 18 19:17:02.719477 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:02.719447    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wpbnx_629ab9c3-0c87-4f52-9770-c861b9ea5705/exporter/0.log"
Sep 18 19:17:02.742087 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:02.742058    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wpbnx_629ab9c3-0c87-4f52-9770-c861b9ea5705/extractor/0.log"
Sep 18 19:17:03.356648 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:03.356610    2558 scope.go:117] "RemoveContainer" containerID="686b3b55e865eb78c50f60ebcfde254767248fe1c87ec9a5f1809c079dd29ada"
Sep 18 19:17:03.370442 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:03.370415    2558 scope.go:117] "RemoveContainer" containerID="405964b1d0d3319dfc506c58efec012d94c0af7da58a5bc8c76c36dafeeeb6d8"
Sep 18 19:17:03.383539 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:03.383512    2558 scope.go:117] "RemoveContainer" containerID="f1ddffa4ba53426284027ccba7c403e79b595ebfff31bf035b568f72caaf7e44"
Sep 18 19:17:05.253326 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:05.253293    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw_98840362-73e4-4d07-9b00-e2f02c2cb44f/istio-proxy/0.log"
Sep 18 19:17:05.374231 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:05.374181    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9cwgw_3b75c261-ec15-4001-a6da-ebef6ce55c62/manager/0.log"
Sep 18 19:17:06.072023 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:06.071987    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-fvbcs_a0b22143-f65e-4cf9-8674-14fe8946d89f/payload-pre-processing/0.log"
Sep 18 19:17:06.098413 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:06.098387    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-bbxqv_ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1/payload-processing/0.log"
Sep 18 19:17:06.120636 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:06.120607    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bd44bd444-kdgsw_7117c313-6fb4-48b0-8f3e-16e2c9c8d551/router/0.log"
Sep 18 19:17:06.131902 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:06.131798    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55bcff78dd-28hzb_839f364a-1d5c-4f8c-88f5-5f54f2d22f55/manager/0.log"
Sep 18 19:17:06.158472 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:06.158447    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55bcff78dd-28hzb_839f364a-1d5c-4f8c-88f5-5f54f2d22f55/copy-manifests/0.log"
Sep 18 19:17:07.586419 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:07.586389    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-5ncqw_98840362-73e4-4d07-9b00-e2f02c2cb44f/istio-proxy/0.log"
Sep 18 19:17:07.713712 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:07.713686    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7cddfd44f8-zdz8r_b1d17e93-f66b-491c-b6d5-42b9473f1caf/manager/0.log"
Sep 18 19:17:07.754739 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:07.754687    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-wzzzd_eba37969-cd6d-4aa6-9307-345a0d5133d9/openshift-lws-operator/0.log"
Sep 18 19:17:08.391522 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:08.391484    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-fvbcs_a0b22143-f65e-4cf9-8674-14fe8946d89f/payload-pre-processing/0.log"
Sep 18 19:17:08.419139 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:08.419100    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-bbxqv_ba97ff99-c0a3-474d-9697-0fb7c0f8dbb1/payload-processing/0.log"
Sep 18 19:17:08.453637 ip-10-0-130-78 kubenswrapper[2558]: I0918 19:17:08.453588    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bd44bd444-kdgsw_7117c313-6fb4-48b0-8f3e-16e2c9c8d551/router/0.log"