Sep 16 07:33:58.959096 ip-10-0-139-143 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 07:33:59.368111 ip-10-0-139-143 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 07:33:59.368111 ip-10-0-139-143 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 07:33:59.368111 ip-10-0-139-143 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 07:33:59.368111 ip-10-0-139-143 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 07:33:59.368111 ip-10-0-139-143 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 07:33:59.369236 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.369154    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 07:33:59.373613 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373597    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 07:33:59.373613 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373613    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373617    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373620    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373623    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373627    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373630    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373632    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373635    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373638    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373641    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373643    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373646    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373649    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373652    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373654    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373657    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373660    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373662    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373665    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373667    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 07:33:59.373678 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373670    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373673    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373675    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373678    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373680    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373683    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373686    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373689    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373694    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373697    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373700    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373703    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373706    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373709    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373712    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373715    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373718    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373720    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373723    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 07:33:59.374213 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373726    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373729    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373731    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373734    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373736    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373739    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373741    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373743    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373746    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373749    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373752    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373754    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373756    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373759    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373761    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373766    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373769    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373771    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373774    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 07:33:59.374665 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373776    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373779    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373781    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373784    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373786    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373789    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373791    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373794    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373796    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373799    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373801    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373804    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373813    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373834    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373837    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373842    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373846    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373850    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373854    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373858    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 07:33:59.375150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373861    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373865    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373869    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373873    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373879    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373885    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373890    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373893    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373896    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373898    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373901    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373904    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373907    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373909    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373912    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373915    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373917    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373920    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373922    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.373925    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 07:33:59.375658 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374605    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374611    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374614    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374617    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374620    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374623    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374626    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374629    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374632    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374634    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374637    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374640    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374642    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374645    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374648    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374650    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374653    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374655    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374658    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374660    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 07:33:59.376158 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374664    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374667    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374670    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374673    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374675    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374678    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374681    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374683    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374686    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374689    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374691    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374694    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374698    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374702    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374704    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374707    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374709    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374712    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374715    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374717    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 07:33:59.376791 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374720    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374722    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374725    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374727    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374730    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374732    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374735    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374737    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374740    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374742    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374745    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374747    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374750    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374753    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374757    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374761    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374764    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374767    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374770    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 07:33:59.377357 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374772    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374775    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374777    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374780    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374783    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374786    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374789    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374791    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374794    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374796    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374799    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374803    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374810    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374813    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374831    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374834    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374837    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374840    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374843    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 07:33:59.377892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374845    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374848    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374850    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374853    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374855    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374858    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374861    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374863    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374866    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374869    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374872    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374875    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374877    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374880    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374883    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374885    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374890    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374893    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374895    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 07:33:59.378429 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374898    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.374900    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375276    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375281    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375285    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375288    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375291    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375294    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375297    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375300    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375303    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375306    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375309    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375312    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375315    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375318    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375320    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375323    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375326    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375329    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 07:33:59.378959 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375332    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375335    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375337    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375340    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375343    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375345    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375348    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375350    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375353    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375357    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375360    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375362    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375365    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375367    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375370    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375373    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375375    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375378    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375380    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375383    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 07:33:59.379505 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375386    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375388    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375391    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375393    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375396    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375398    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375401    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375405    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375409    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375411    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375414    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375417    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375420    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375423    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375425    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375428    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375430    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375432    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375436    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 07:33:59.380116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375438    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375441    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375445    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375448    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375450    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375453    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375455    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375458    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375460    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375463    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375465    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375468    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375470    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375475    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375477    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375479    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375482    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375484    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375486    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375489    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 07:33:59.380636 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375491    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375494    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375496    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375500    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375504    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375507    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375509    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375512    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375515    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375518    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375520    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375523    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375525    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375528    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375531    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375535    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375538    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375541    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375545    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 07:33:59.381165 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375547    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375550    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.375552    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375620    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375652    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375659    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375663    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375667    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375671    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375675    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375680    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375683    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375686    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375689    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375692    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375696    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375699    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375702    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375705    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375708    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375711    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375715    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375717    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 07:33:59.381631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375721    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375724    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375727    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375731    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375734    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375737    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375741    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375744    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375747    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375750    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375753    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375756    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375760    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375763    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375765    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375768    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375771    2581 flags.go:64] FLAG: --enable-server="true"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375775    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375779    2581 flags.go:64] FLAG: --event-burst="100"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375783    2581 flags.go:64] FLAG: --event-qps="50"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375785    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375788    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375791    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375795    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 07:33:59.382205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375798    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375801    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375804    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375806    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375810    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375812    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375828    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375831    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375834    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375837    2581 flags.go:64] FLAG: --feature-gates=""
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375841    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375844    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375847    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375850    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375852    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375855    2581 flags.go:64] FLAG: --help="false"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375858    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375861    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375864    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375867    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375870    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375873    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375876    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 07:33:59.382805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375879    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375881    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375886    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375889    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375892    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375895    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375898    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375901    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375904    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375907    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375910    2581 flags.go:64] FLAG: --lock-file=""
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375912    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375915    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375918    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375923    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375926    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375929    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375932    2581 flags.go:64] FLAG: --logging-format="text"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375935    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375939    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375942    2581 flags.go:64] FLAG: --manifest-url=""
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375944    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375948    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375951    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375955    2581 flags.go:64] FLAG: --max-pods="110"
Sep 16 07:33:59.383407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375958    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375961    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375964    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375967    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375970    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375973    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375976    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375982    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375985    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375990    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375993    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.375997    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376002    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376005    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376008    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376011    2581 flags.go:64] FLAG: --port="10250"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376014    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376017    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f6dc1af12746e695"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376020    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376023    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376026    2581 flags.go:64] FLAG: --register-node="true"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376029    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376033    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376036    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 16 07:33:59.384046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376039    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376041    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376045    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376048    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376051    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376053    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376056    2581 flags.go:64] FLAG: --runonce="false"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376059    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376062    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376065    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376068    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376070    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376073    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376076    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376079    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376082    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376085    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376088    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376093    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376097    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376100    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376103    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376108    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376111    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376114    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 07:33:59.384666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376117    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376120    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376123    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376125    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376128    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376132    2581 flags.go:64] FLAG: --v="2"
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376136    2581 flags.go:64] FLAG: --version="false"
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376140    2581 flags.go:64] FLAG: --vmodule=""
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376144    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.376147    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376268    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376272    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376275    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376278    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376281    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376284    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376286    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376290    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376292    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376296    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376300    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 07:33:59.385291 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376303    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376306    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376309    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376312    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376316    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376319    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376321    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376324    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376327    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376330    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376333    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376335    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376338    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376341    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376344    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376354    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376358    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376361    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376363    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 07:33:59.385805 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376366    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376380    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376384    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376388    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376391    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376394    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376397    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376400    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376403    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376406    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376408    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376411    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376413    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376416    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376419    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376421    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376424    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376427    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376430    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376433    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 07:33:59.386311 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376436    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376439    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376441    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376444    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376447    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376449    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376452    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376455    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376458    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376461    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376464    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376466    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376469    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376472    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376474    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376477    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376480    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376482    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376485    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 07:33:59.386796 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376487    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376490    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376494    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376497    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376500    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376502    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376505    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376508    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376510    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376513    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376517    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376519    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376522    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376525    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376527    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376530    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376533    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376535    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376538    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 07:33:59.387272 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376541    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 07:33:59.387741 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376543    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 07:33:59.387741 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376546    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 07:33:59.387741 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376550    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 07:33:59.387741 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376553    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 07:33:59.387741 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376555    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 07:33:59.387741 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376558    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 07:33:59.387741 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376560    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 07:33:59.387741 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376563    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 07:33:59.387741 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376566    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 07:33:59.387741 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.376568    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 07:33:59.387741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.377085    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 07:33:59.387741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.384317    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 07:33:59.387741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.384333    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 07:33:59.387741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.385247    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 07:33:59.387741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.385254    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385329    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385335    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385338    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385342    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385346    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385349    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385352    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385355    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385358    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385360    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385363    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385366    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385370    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385373    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385376    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385378    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385381    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385384    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385386    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 07:33:59.388157 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385389    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385391    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385394    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385397    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385400    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385404    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385407    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385410    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385413    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385416    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385419    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385422    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385425    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385427    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385430    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385433    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385436    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385438    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385441    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 07:33:59.388956 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385443    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385446    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385449    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385452    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385455    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385458    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385460    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385469    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385472    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385474    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385477    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385480    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385483    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385485    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385488    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385490    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385493    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385496    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385499    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385501    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 07:33:59.389622 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385504    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385507    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385511    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385515    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385519    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385521    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385524    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385527    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385530    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385533    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385536    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385539    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385541    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385544    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385547    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385550    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385554    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385557    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385560    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 07:33:59.390128 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385569    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385580    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385583    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385585    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385588    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385591    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385593    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385596    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385598    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385601    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385604    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385606    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385609    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385612    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385614    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385617    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385620    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385623    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385626    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385628    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 07:33:59.390724 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385631    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 07:33:59.391233 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385634    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 07:33:59.391233 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.385639    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 07:33:59.391233 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385837    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 07:33:59.391233 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385842    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 07:33:59.391233 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385846    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 07:33:59.391233 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385849    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 07:33:59.391233 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385852    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 07:33:59.391233 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385855    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 07:33:59.391233 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385858    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 07:33:59.391233 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385861    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 07:33:59.391233 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385864    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 07:33:59.391233 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385867    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 07:33:59.391233 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385870    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 07:33:59.391233 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385878    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 07:33:59.391233 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385881    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385884    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385886    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385889    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385892    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385894    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385897    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385899    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385902    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385905    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385907    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385911    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385915    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385918    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385921    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385924    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385926    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385929    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385932    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 07:33:59.391612 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385936    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385939    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385943    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385946    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385949    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385952    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385955    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385958    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385961    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385964    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385967    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385969    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385972    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385980    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385983    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385985    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385988    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385991    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385993    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 07:33:59.392094 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385996    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.385998    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386001    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386004    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386006    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386009    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386012    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386014    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386017    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386020    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386022    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386025    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386028    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386030    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386033    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386036    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386038    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386041    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386044    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386046    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 07:33:59.392544 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386049    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386052    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386054    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386058    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386062    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386064    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386067    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386075    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386078    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386081    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386084    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386086    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386089    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386091    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386094    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386096    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386099    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386101    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386104    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 07:33:59.393027 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386107    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 07:33:59.393541 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386110    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 07:33:59.393541 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386112    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 07:33:59.393541 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386115    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 07:33:59.393541 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386117    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 07:33:59.393541 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386120    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 07:33:59.393541 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386122    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 07:33:59.393541 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386125    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 07:33:59.393541 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386127    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 07:33:59.393541 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:33:59.386130    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 07:33:59.393541 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.386135    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 07:33:59.393541 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.386242    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 07:33:59.393541 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.388130    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 07:33:59.393541 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.393221    2581 server.go:1019] "Starting client certificate rotation"
Sep 16 07:33:59.393541 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.393323    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 07:33:59.393909 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.393383    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 07:33:59.395720 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.395701    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 07:33:59.396864 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.396844    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 07:33:59.408058 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.408043    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 16 07:33:59.411645 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.411630    2581 log.go:25] "Validated CRI v1 image API"
Sep 16 07:33:59.412598 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.412576    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 07:33:59.415795 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.415776    2581 fs.go:135] Filesystem UUIDs: map[48f8af98-d965-4b83-bab3-0b23cfc99a84:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cd65c768-fa7f-47a0-a308-896e2e02a782:/dev/nvme0n1p3]
Sep 16 07:33:59.415873 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.415794    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 07:33:59.421362 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.421344    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 07:33:59.421496 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.421344    2581 manager.go:216] Machine: {Timestamp:2026-09-16 07:33:59.420138102 +0000 UTC m=+0.346750247 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100922 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f26c9bfaf8150db7ca39c2a430e79 SystemUUID:ec2f26c9-bfaf-8150-db7c-a39c2a430e79 BootID:bf0d5ed5-e0aa-4213-a444-4b6f2a9c30eb Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:da:14:4e:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:da:14:4e:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:f3:b1:9d:fd:e4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 07:33:59.421595 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.421498    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 07:33:59.421626 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.421590    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 07:33:59.425073 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.425051    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 07:33:59.425213 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.425083    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-143.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 07:33:59.425838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.425828    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 07:33:59.425876 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.425841    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 07:33:59.425876 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.425848    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 07:33:59.426408 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.426398    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 07:33:59.427166 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.427157    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 07:33:59.430708 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.430696    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 07:33:59.430835 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.430813    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 07:33:59.430872 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.430861    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 07:33:59.430904 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.430873    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 07:33:59.430904 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.430885    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 07:33:59.430904 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.430901    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 16 07:33:59.430985 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.430910    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 07:33:59.431264 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.431251    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 07:33:59.431310 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.431274    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 07:33:59.435751 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.435732    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 07:33:59.438165 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.438147    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 07:33:59.438253 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.438185    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 07:33:59.439639 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.439624    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 07:33:59.439706 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.439644    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 07:33:59.439706 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.439662    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 07:33:59.439706 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.439671    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 07:33:59.439706 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.439683    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 07:33:59.439706 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.439691    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 07:33:59.439910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.439716    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 07:33:59.439910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.439725    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 07:33:59.439910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.439735    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 07:33:59.439910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.439743    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 07:33:59.439910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.439756    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 07:33:59.439910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.439765    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 07:33:59.440103 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.440082    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jqgsm"
Sep 16 07:33:59.440596 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.440584    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 07:33:59.440642 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.440598    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 07:33:59.441360 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.441335    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 07:33:59.441443 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.441374    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 07:33:59.445095 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.445084    2581 server.go:1268] "Started kubelet"
Sep 16 07:33:59.445200 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.445154    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 07:33:59.445259 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.445187    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 07:33:59.445259 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.445254    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 07:33:59.445973 ip-10-0-139-143 systemd[1]: Started Kubernetes Kubelet.
Sep 16 07:33:59.447149 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.447132    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 07:33:59.447407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.447389    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jqgsm"
Sep 16 07:33:59.447898 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.447886    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 07:33:59.453050 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.453032    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-143.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 07:33:59.453453 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.453433    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 07:33:59.453783 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.453769    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 07:33:59.454098 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.453048    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-143.ec2.internal.18d5bcd28d0a52a9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-143.ec2.internal,UID:ip-10-0-139-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-143.ec2.internal,},FirstTimestamp:2026-09-16 07:33:59.445058217 +0000 UTC m=+0.371670360,LastTimestamp:2026-09-16 07:33:59.445058217 +0000 UTC m=+0.371670360,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-143.ec2.internal,}"
Sep 16 07:33:59.454813 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.454798    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 07:33:59.454967 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.454950    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Sep 16 07:33:59.455566 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.455550    2581 factory.go:55] Registering systemd factory
Sep 16 07:33:59.455566 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.455566    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 16 07:33:59.455774 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.455760    2581 factory.go:153] Registering CRI-O factory
Sep 16 07:33:59.455842 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.455778    2581 factory.go:223] Registration of the crio container factory successfully
Sep 16 07:33:59.455878 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.455843    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 07:33:59.455878 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.455863    2581 factory.go:103] Registering Raw factory
Sep 16 07:33:59.455878 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.455877    2581 manager.go:1196] Started watching for new ooms in manager
Sep 16 07:33:59.455986 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.455939    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 07:33:59.455986 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.455957    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 07:33:59.456072 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.455935    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 07:33:59.456158 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.456146    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 07:33:59.456202 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.456160    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 07:33:59.456839 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.456810    2581 manager.go:349] Starting recovery of all containers
Sep 16 07:33:59.460171 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.460147    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 07:33:59.464136 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.464108    2581 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-143.ec2.internal\" not found" node="ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.470853 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.470646    2581 manager.go:354] Recovery completed
Sep 16 07:33:59.475886 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.475873    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 07:33:59.477612 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.477599    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 07:33:59.477665 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.477626    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 07:33:59.477665 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.477639    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientPID"
Sep 16 07:33:59.478105 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.478093    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 07:33:59.478105 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.478100    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 07:33:59.478105 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.478104    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 07:33:59.478239 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.478118    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 07:33:59.478807 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.478793    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 07:33:59.478929 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.478918    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 07:33:59.478976 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.478936    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientPID"
Sep 16 07:33:59.555916 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.555886    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Sep 16 07:33:59.621704 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.621607    2581 policy_none.go:49] "None policy: Start"
Sep 16 07:33:59.621704 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.621631    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 07:33:59.621704 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.621644    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 07:33:59.622929 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.622916    2581 policy_none.go:47] "Start"
Sep 16 07:33:59.650030 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.646458    2581 manager.go:337] "Starting Device Plugin manager"
Sep 16 07:33:59.650030 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.646490    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 07:33:59.650030 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.646500    2581 server.go:87] "Starting device plugin registration server"
Sep 16 07:33:59.650030 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.646811    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 07:33:59.650030 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.646838    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 07:33:59.650030 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.646957    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 07:33:59.650030 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.647038    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 07:33:59.650030 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.647046    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 07:33:59.650030 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.647602    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 07:33:59.650030 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.647654    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-143.ec2.internal\" not found"
Sep 16 07:33:59.675896 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.675868    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 07:33:59.677020 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.676997    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 07:33:59.677101 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.677059    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 07:33:59.677101 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.677084    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 07:33:59.677184 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.677123    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 07:33:59.679262 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.679241    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 07:33:59.747022 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.746984    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 07:33:59.747876 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.747862    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 07:33:59.747942 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.747890    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 07:33:59.747942 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.747899    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientPID"
Sep 16 07:33:59.747942 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.747923    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.755291 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.755275    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.755385 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.755300    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-143.ec2.internal\": node \"ip-10-0-139-143.ec2.internal\" not found"
Sep 16 07:33:59.771914 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.771893    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Sep 16 07:33:59.777968 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.777948    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal"]
Sep 16 07:33:59.778039 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.778018    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 07:33:59.778747 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.778733    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 07:33:59.778795 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.778759    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 07:33:59.778795 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.778769    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientPID"
Sep 16 07:33:59.780561 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.780549    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 07:33:59.780681 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.780667    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.780721 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.780696    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 07:33:59.781197 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.781182    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 07:33:59.781265 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.781208    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 07:33:59.781265 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.781218    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientPID"
Sep 16 07:33:59.781351 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.781267    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 07:33:59.781351 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.781293    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 07:33:59.781351 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.781308    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientPID"
Sep 16 07:33:59.782485 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.782472    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.782547 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.782495    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 07:33:59.783175 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.783155    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 07:33:59.783247 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.783180    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 07:33:59.783247 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.783193    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientPID"
Sep 16 07:33:59.811350 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.811330    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-143.ec2.internal\" not found" node="ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.815867 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.815851    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-143.ec2.internal\" not found" node="ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.858743 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.858722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62629d7452b8a7ebac926f07ec860193-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal\" (UID: \"62629d7452b8a7ebac926f07ec860193\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.858861 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.858750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62629d7452b8a7ebac926f07ec860193-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal\" (UID: \"62629d7452b8a7ebac926f07ec860193\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.858861 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.858775    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c1a7e92e9ab0a02d77b710fd18e6783c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-143.ec2.internal\" (UID: \"c1a7e92e9ab0a02d77b710fd18e6783c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.872877 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.872841    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Sep 16 07:33:59.959055 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.959028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62629d7452b8a7ebac926f07ec860193-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal\" (UID: \"62629d7452b8a7ebac926f07ec860193\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.959055 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.959041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62629d7452b8a7ebac926f07ec860193-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal\" (UID: \"62629d7452b8a7ebac926f07ec860193\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.959203 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.959080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62629d7452b8a7ebac926f07ec860193-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal\" (UID: \"62629d7452b8a7ebac926f07ec860193\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.959203 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.959105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c1a7e92e9ab0a02d77b710fd18e6783c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-143.ec2.internal\" (UID: \"c1a7e92e9ab0a02d77b710fd18e6783c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.959203 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.959144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c1a7e92e9ab0a02d77b710fd18e6783c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-143.ec2.internal\" (UID: \"c1a7e92e9ab0a02d77b710fd18e6783c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.959203 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:33:59.959162    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62629d7452b8a7ebac926f07ec860193-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal\" (UID: \"62629d7452b8a7ebac926f07ec860193\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Sep 16 07:33:59.973105 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:33:59.973084    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Sep 16 07:34:00.073648 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:00.073618    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Sep 16 07:34:00.114783 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.114764    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Sep 16 07:34:00.118848 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.118685    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal"
Sep 16 07:34:00.174427 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:00.174367    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Sep 16 07:34:00.275047 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:00.275002    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Sep 16 07:34:00.279710 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.279691    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 07:34:00.356053 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.356023    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal"
Sep 16 07:34:00.366583 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.366564    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 07:34:00.368365 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.368352    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Sep 16 07:34:00.379526 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.379509    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 07:34:00.390990 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.390971    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 07:34:00.391107 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.391090    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 07:34:00.391150 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.391139    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 07:34:00.391185 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.391145    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 07:34:00.433226 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.433179    2581 apiserver.go:52] "Watching apiserver"
Sep 16 07:34:00.450907 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.450878    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 07:34:00.451450 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.451427    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-xlmf2","openshift-image-registry/node-ca-zrs62","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal","openshift-multus/multus-additional-cni-plugins-qvcgx","openshift-multus/multus-h8wqx","openshift-ovn-kubernetes/ovnkube-node-gsjsp","kube-system/konnectivity-agent-fsz9s","kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq","openshift-cluster-node-tuning-operator/tuned-gsfxp","openshift-dns/node-resolver-pw9bz","openshift-multus/network-metrics-daemon-hrrnm","openshift-network-diagnostics/network-check-target-h4pbq","openshift-network-operator/iptables-alerter-dfg8w"]
Sep 16 07:34:00.453039 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.453024    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fsz9s"
Sep 16 07:34:00.454193 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.454035    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 07:34:00.454193 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.454075    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 07:29:00 +0000 UTC" deadline="2028-04-03 05:38:36.895824931 +0000 UTC"
Sep 16 07:34:00.454193 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.454148    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13558h4m36.441681823s"
Sep 16 07:34:00.455949 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.455904    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 07:34:00.456028 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.455989    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:00.456028 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.456008    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 07:34:00.456132 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:00.456059    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:00.456307 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.456267    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7k7p6\""
Sep 16 07:34:00.457594 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.457573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dfg8w"
Sep 16 07:34:00.458794 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.458778    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.459968 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.459938    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 07:34:00.460077 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.459993    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 07:34:00.460077 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.460014    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bmsbq\""
Sep 16 07:34:00.460077 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.460016    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 07:34:00.460275 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.460208    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.460938 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.460921    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 07:34:00.461085 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.461070    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 07:34:00.461320 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.461303    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.461320 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.461317    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 07:34:00.461458 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.461394    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 07:34:00.461458 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.461404    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dccdq\""
Sep 16 07:34:00.461717 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.461681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76e718e8-9332-436d-b35b-89253c6710a7-konnectivity-ca\") pod \"konnectivity-agent-fsz9s\" (UID: \"76e718e8-9332-436d-b35b-89253c6710a7\") " pod="kube-system/konnectivity-agent-fsz9s"
Sep 16 07:34:00.461803 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.461724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh5tq\" (UniqueName: \"kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq\") pod \"network-check-target-h4pbq\" (UID: \"8cdad5a8-3714-41e1-8f19-bd579800dc49\") " pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:00.461803 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.461741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d93c2061-f013-4be5-8520-ce0fe0f89c04-iptables-alerter-script\") pod \"iptables-alerter-dfg8w\" (UID: \"d93c2061-f013-4be5-8520-ce0fe0f89c04\") " pod="openshift-network-operator/iptables-alerter-dfg8w"
Sep 16 07:34:00.461803 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.461758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76e718e8-9332-436d-b35b-89253c6710a7-agent-certs\") pod \"konnectivity-agent-fsz9s\" (UID: \"76e718e8-9332-436d-b35b-89253c6710a7\") " pod="kube-system/konnectivity-agent-fsz9s"
Sep 16 07:34:00.461803 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.461799    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d93c2061-f013-4be5-8520-ce0fe0f89c04-host-slash\") pod \"iptables-alerter-dfg8w\" (UID: \"d93c2061-f013-4be5-8520-ce0fe0f89c04\") " pod="openshift-network-operator/iptables-alerter-dfg8w"
Sep 16 07:34:00.462061 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.461828    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jjr8\" (UniqueName: \"kubernetes.io/projected/d93c2061-f013-4be5-8520-ce0fe0f89c04-kube-api-access-4jjr8\") pod \"iptables-alerter-dfg8w\" (UID: \"d93c2061-f013-4be5-8520-ce0fe0f89c04\") " pod="openshift-network-operator/iptables-alerter-dfg8w"
Sep 16 07:34:00.462061 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.461970    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 07:34:00.462579 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.462564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.462674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.462596    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 07:34:00.462674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.462632    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sgvqd\""
Sep 16 07:34:00.463807 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.463789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:00.463883 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:00.463858    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:00.464250 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.464233    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 07:34:00.465104 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.465050    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zrs62"
Sep 16 07:34:00.465204 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.465116    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 07:34:00.465204 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.465122    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 07:34:00.465204 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.465146    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 07:34:00.465472 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.465458    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 07:34:00.466340 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.466325    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.466423 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.466378    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 07:34:00.466423 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.466396    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xl8dw\""
Sep 16 07:34:00.466530 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.466421    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lfzwn\""
Sep 16 07:34:00.466530 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.466445    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 07:34:00.466756 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.466741    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 07:34:00.466818 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.466760    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 07:34:00.467632 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.467620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pw9bz"
Sep 16 07:34:00.467858 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.467826    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4tdj5\""
Sep 16 07:34:00.467991 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.467857    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 07:34:00.468048 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.468010    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 07:34:00.468096 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.468053    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 07:34:00.468962 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.468946    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7gnkh\""
Sep 16 07:34:00.469159 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.469135    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:00.469238 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:00.469217    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:00.469297 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.469251    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 07:34:00.469355 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.469301    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 07:34:00.469355 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.469323    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 07:34:00.470052 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.470035    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 07:34:00.470132 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.470037    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2pvw7\""
Sep 16 07:34:00.470188 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.470169    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 07:34:00.517984 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.517960    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4cp65"
Sep 16 07:34:00.530580 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.530565    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4cp65"
Sep 16 07:34:00.557137 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.557116    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 07:34:00.562916 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.562895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1b6df6f-8ab5-4abb-87b9-8947b5b36043-tmp-dir\") pod \"node-resolver-pw9bz\" (UID: \"a1b6df6f-8ab5-4abb-87b9-8947b5b36043\") " pod="openshift-dns/node-resolver-pw9bz"
Sep 16 07:34:00.563015 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.562923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46de41b3-6907-4352-81cf-1216677e98c7-cnibin\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.563015 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.562943    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs\") pod \"network-metrics-daemon-hrrnm\" (UID: \"3b0fd941-35fa-4dac-a59e-38cbea3386d5\") " pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:00.563015 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.562959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-var-lib-cni-multus\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.563015 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.562974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-kubelet\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.563015 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.562988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-run-systemd\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.563282 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46de41b3-6907-4352-81cf-1216677e98c7-system-cni-dir\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.563282 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76e718e8-9332-436d-b35b-89253c6710a7-konnectivity-ca\") pod \"konnectivity-agent-fsz9s\" (UID: \"76e718e8-9332-436d-b35b-89253c6710a7\") " pod="kube-system/konnectivity-agent-fsz9s"
Sep 16 07:34:00.563282 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563126    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-cnibin\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.563282 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njxj2\" (UniqueName: \"kubernetes.io/projected/6aeb9bac-7199-48cc-8159-16017f479f7e-kube-api-access-njxj2\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.563282 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-host\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.563282 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1b6df6f-8ab5-4abb-87b9-8947b5b36043-hosts-file\") pod \"node-resolver-pw9bz\" (UID: \"a1b6df6f-8ab5-4abb-87b9-8947b5b36043\") " pod="openshift-dns/node-resolver-pw9bz"
Sep 16 07:34:00.563547 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d93c2061-f013-4be5-8520-ce0fe0f89c04-host-slash\") pod \"iptables-alerter-dfg8w\" (UID: \"d93c2061-f013-4be5-8520-ce0fe0f89c04\") " pod="openshift-network-operator/iptables-alerter-dfg8w"
Sep 16 07:34:00.563547 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-hostroot\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.563547 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-multus-conf-dir\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.563547 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563408    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d93c2061-f013-4be5-8520-ce0fe0f89c04-host-slash\") pod \"iptables-alerter-dfg8w\" (UID: \"d93c2061-f013-4be5-8520-ce0fe0f89c04\") " pod="openshift-network-operator/iptables-alerter-dfg8w"
Sep 16 07:34:00.563547 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563430    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-etc-selinux\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.563547 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-sysconfig\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.563547 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-lib-modules\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.563547 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46de41b3-6907-4352-81cf-1216677e98c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.563547 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/46de41b3-6907-4352-81cf-1216677e98c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.563927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wtqk\" (UniqueName: \"kubernetes.io/projected/46de41b3-6907-4352-81cf-1216677e98c7-kube-api-access-9wtqk\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.563927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-node-log\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.563927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76e718e8-9332-436d-b35b-89253c6710a7-konnectivity-ca\") pod \"konnectivity-agent-fsz9s\" (UID: \"76e718e8-9332-436d-b35b-89253c6710a7\") " pod="kube-system/konnectivity-agent-fsz9s"
Sep 16 07:34:00.563927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.563927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-registration-dir\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.563927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zqqn\" (UniqueName: \"kubernetes.io/projected/d867daeb-9f34-4224-91e7-65889fc3e051-kube-api-access-2zqqn\") pod \"node-ca-zrs62\" (UID: \"d867daeb-9f34-4224-91e7-65889fc3e051\") " pod="openshift-image-registry/node-ca-zrs62"
Sep 16 07:34:00.563927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-run-multus-certs\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.563927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563812    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-etc-openvswitch\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.563927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-run-openvswitch\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.563927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-multus-cni-dir\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.563927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6aeb9bac-7199-48cc-8159-16017f479f7e-cni-binary-copy\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.563927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.563927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dfd2dab-c07b-44bf-b150-7e295cd0794c-ovnkube-config\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.563927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dfd2dab-c07b-44bf-b150-7e295cd0794c-ovn-node-metrics-cert\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n6wq\" (UniqueName: \"kubernetes.io/projected/2dfd2dab-c07b-44bf-b150-7e295cd0794c-kube-api-access-8n6wq\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563956    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a73e698d-df57-4277-9435-f7d5bfc3be6f-kubelet-config\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.563993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-sysctl-d\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-run-k8s-cni-cncf-io\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-run-netns\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-cni-bin\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-modprobe-d\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564149    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-systemd\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-tuned\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jjr8\" (UniqueName: \"kubernetes.io/projected/d93c2061-f013-4be5-8520-ce0fe0f89c04-kube-api-access-4jjr8\") pod \"iptables-alerter-dfg8w\" (UID: \"d93c2061-f013-4be5-8520-ce0fe0f89c04\") " pod="openshift-network-operator/iptables-alerter-dfg8w"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d92t6\" (UniqueName: \"kubernetes.io/projected/3b0fd941-35fa-4dac-a59e-38cbea3386d5-kube-api-access-d92t6\") pod \"network-metrics-daemon-hrrnm\" (UID: \"3b0fd941-35fa-4dac-a59e-38cbea3386d5\") " pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-multus-socket-dir-parent\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-slash\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-log-socket\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dfd2dab-c07b-44bf-b150-7e295cd0794c-ovnkube-script-lib\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.564555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-socket-dir\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-device-dir\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-system-cni-dir\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564485    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a73e698d-df57-4277-9435-f7d5bfc3be6f-dbus\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac35ebd4-906b-44ad-b7c3-9be83389def6-tmp\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46de41b3-6907-4352-81cf-1216677e98c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d867daeb-9f34-4224-91e7-65889fc3e051-host\") pod \"node-ca-zrs62\" (UID: \"d867daeb-9f34-4224-91e7-65889fc3e051\") " pod="openshift-image-registry/node-ca-zrs62"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564603    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-var-lib-cni-bin\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-cni-netd\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-kubernetes\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564688    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxqcb\" (UniqueName: \"kubernetes.io/projected/a1b6df6f-8ab5-4abb-87b9-8947b5b36043-kube-api-access-sxqcb\") pod \"node-resolver-pw9bz\" (UID: \"a1b6df6f-8ab5-4abb-87b9-8947b5b36043\") " pod="openshift-dns/node-resolver-pw9bz"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/46de41b3-6907-4352-81cf-1216677e98c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d93c2061-f013-4be5-8520-ce0fe0f89c04-iptables-alerter-script\") pod \"iptables-alerter-dfg8w\" (UID: \"d93c2061-f013-4be5-8520-ce0fe0f89c04\") " pod="openshift-network-operator/iptables-alerter-dfg8w"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-var-lib-kubelet\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.565187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564888    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn7xq\" (UniqueName: \"kubernetes.io/projected/ac35ebd4-906b-44ad-b7c3-9be83389def6-kube-api-access-kn7xq\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d867daeb-9f34-4224-91e7-65889fc3e051-serviceca\") pod \"node-ca-zrs62\" (UID: \"d867daeb-9f34-4224-91e7-65889fc3e051\") " pod="openshift-image-registry/node-ca-zrs62"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.564964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6aeb9bac-7199-48cc-8159-16017f479f7e-multus-daemon-config\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-run-netns\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565047    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-var-lib-openvswitch\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565071    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dfd2dab-c07b-44bf-b150-7e295cd0794c-env-overrides\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqc56\" (UniqueName: \"kubernetes.io/projected/cac2eb90-87d1-4c96-aede-34bc47a5b958-kube-api-access-sqc56\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh5tq\" (UniqueName: \"kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq\") pod \"network-check-target-h4pbq\" (UID: \"8cdad5a8-3714-41e1-8f19-bd579800dc49\") " pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565154    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-sys\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76e718e8-9332-436d-b35b-89253c6710a7-agent-certs\") pod \"konnectivity-agent-fsz9s\" (UID: \"76e718e8-9332-436d-b35b-89253c6710a7\") " pod="kube-system/konnectivity-agent-fsz9s"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-systemd-units\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-run-ovn\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-var-lib-kubelet\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-etc-kubernetes\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-sys-fs\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565375    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d93c2061-f013-4be5-8520-ce0fe0f89c04-iptables-alerter-script\") pod \"iptables-alerter-dfg8w\" (UID: \"d93c2061-f013-4be5-8520-ce0fe0f89c04\") " pod="openshift-network-operator/iptables-alerter-dfg8w"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-sysctl-conf\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.565766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-run\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.566375 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.565613    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 07:34:00.568738 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.568722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76e718e8-9332-436d-b35b-89253c6710a7-agent-certs\") pod \"konnectivity-agent-fsz9s\" (UID: \"76e718e8-9332-436d-b35b-89253c6710a7\") " pod="kube-system/konnectivity-agent-fsz9s"
Sep 16 07:34:00.576834 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:00.576815    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 07:34:00.576972 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:00.576839    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 07:34:00.576972 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:00.576850    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vh5tq for pod openshift-network-diagnostics/network-check-target-h4pbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 07:34:00.576972 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:00.576929    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq podName:8cdad5a8-3714-41e1-8f19-bd579800dc49 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:01.076909757 +0000 UTC m=+2.003521904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vh5tq" (UniqueName: "kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq") pod "network-check-target-h4pbq" (UID: "8cdad5a8-3714-41e1-8f19-bd579800dc49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 07:34:00.581738 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.581716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jjr8\" (UniqueName: \"kubernetes.io/projected/d93c2061-f013-4be5-8520-ce0fe0f89c04-kube-api-access-4jjr8\") pod \"iptables-alerter-dfg8w\" (UID: \"d93c2061-f013-4be5-8520-ce0fe0f89c04\") " pod="openshift-network-operator/iptables-alerter-dfg8w"
Sep 16 07:34:00.665808 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.665775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-var-lib-cni-bin\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.665808 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.665809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-cni-netd\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.665999 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.665841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-kubernetes\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.665999 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.665887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-var-lib-cni-bin\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.665999 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.665900    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-cni-netd\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.665999 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.665927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxqcb\" (UniqueName: \"kubernetes.io/projected/a1b6df6f-8ab5-4abb-87b9-8947b5b36043-kube-api-access-sxqcb\") pod \"node-resolver-pw9bz\" (UID: \"a1b6df6f-8ab5-4abb-87b9-8947b5b36043\") " pod="openshift-dns/node-resolver-pw9bz"
Sep 16 07:34:00.665999 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.665954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-kubernetes\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.665999 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.665959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/46de41b3-6907-4352-81cf-1216677e98c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.665999 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.665994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-var-lib-kubelet\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.666303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn7xq\" (UniqueName: \"kubernetes.io/projected/ac35ebd4-906b-44ad-b7c3-9be83389def6-kube-api-access-kn7xq\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.666303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d867daeb-9f34-4224-91e7-65889fc3e051-serviceca\") pod \"node-ca-zrs62\" (UID: \"d867daeb-9f34-4224-91e7-65889fc3e051\") " pod="openshift-image-registry/node-ca-zrs62"
Sep 16 07:34:00.666303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6aeb9bac-7199-48cc-8159-16017f479f7e-multus-daemon-config\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.666303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-run-netns\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.666303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666128    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-var-lib-kubelet\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.666303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-var-lib-openvswitch\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.666303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dfd2dab-c07b-44bf-b150-7e295cd0794c-env-overrides\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.666303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-var-lib-openvswitch\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.666303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqc56\" (UniqueName: \"kubernetes.io/projected/cac2eb90-87d1-4c96-aede-34bc47a5b958-kube-api-access-sqc56\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.666303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666203    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-run-netns\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.666303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-sys\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.666303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-systemd-units\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666315    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-sys\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-run-ovn\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-systemd-units\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-var-lib-kubelet\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-etc-kubernetes\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-run-ovn\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-sys-fs\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-sysctl-conf\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-var-lib-kubelet\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-etc-kubernetes\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666500    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-run\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666508    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-sys-fs\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1b6df6f-8ab5-4abb-87b9-8947b5b36043-tmp-dir\") pod \"node-resolver-pw9bz\" (UID: \"a1b6df6f-8ab5-4abb-87b9-8947b5b36043\") " pod="openshift-dns/node-resolver-pw9bz"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46de41b3-6907-4352-81cf-1216677e98c7-cnibin\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-run\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666571    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-sysctl-conf\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs\") pod \"network-metrics-daemon-hrrnm\" (UID: \"3b0fd941-35fa-4dac-a59e-38cbea3386d5\") " pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:00.666930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/46de41b3-6907-4352-81cf-1216677e98c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-var-lib-cni-multus\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46de41b3-6907-4352-81cf-1216677e98c7-cnibin\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d867daeb-9f34-4224-91e7-65889fc3e051-serviceca\") pod \"node-ca-zrs62\" (UID: \"d867daeb-9f34-4224-91e7-65889fc3e051\") " pod="openshift-image-registry/node-ca-zrs62"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-kubelet\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-kubelet\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dfd2dab-c07b-44bf-b150-7e295cd0794c-env-overrides\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-var-lib-cni-multus\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-run-systemd\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46de41b3-6907-4352-81cf-1216677e98c7-system-cni-dir\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-run-systemd\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-cnibin\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:00.666782    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njxj2\" (UniqueName: \"kubernetes.io/projected/6aeb9bac-7199-48cc-8159-16017f479f7e-kube-api-access-njxj2\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666794    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46de41b3-6907-4352-81cf-1216677e98c7-system-cni-dir\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-host\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6aeb9bac-7199-48cc-8159-16017f479f7e-multus-daemon-config\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.667841 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:00.666846    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs podName:3b0fd941-35fa-4dac-a59e-38cbea3386d5 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:01.166827775 +0000 UTC m=+2.093439927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs") pod "network-metrics-daemon-hrrnm" (UID: "3b0fd941-35fa-4dac-a59e-38cbea3386d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-host\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1b6df6f-8ab5-4abb-87b9-8947b5b36043-tmp-dir\") pod \"node-resolver-pw9bz\" (UID: \"a1b6df6f-8ab5-4abb-87b9-8947b5b36043\") " pod="openshift-dns/node-resolver-pw9bz"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1b6df6f-8ab5-4abb-87b9-8947b5b36043-hosts-file\") pod \"node-resolver-pw9bz\" (UID: \"a1b6df6f-8ab5-4abb-87b9-8947b5b36043\") " pod="openshift-dns/node-resolver-pw9bz"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666940    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-hostroot\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-cnibin\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-multus-conf-dir\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.666995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-hostroot\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1b6df6f-8ab5-4abb-87b9-8947b5b36043-hosts-file\") pod \"node-resolver-pw9bz\" (UID: \"a1b6df6f-8ab5-4abb-87b9-8947b5b36043\") " pod="openshift-dns/node-resolver-pw9bz"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-etc-selinux\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-sysconfig\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667036    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-multus-conf-dir\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667078    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-etc-selinux\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667101    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-sysconfig\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-lib-modules\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46de41b3-6907-4352-81cf-1216677e98c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-lib-modules\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/46de41b3-6907-4352-81cf-1216677e98c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.668711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wtqk\" (UniqueName: \"kubernetes.io/projected/46de41b3-6907-4352-81cf-1216677e98c7-kube-api-access-9wtqk\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-node-log\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-registration-dir\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zqqn\" (UniqueName: \"kubernetes.io/projected/d867daeb-9f34-4224-91e7-65889fc3e051-kube-api-access-2zqqn\") pod \"node-ca-zrs62\" (UID: \"d867daeb-9f34-4224-91e7-65889fc3e051\") " pod="openshift-image-registry/node-ca-zrs62"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667325    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-node-log\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667364    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-registration-dir\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-run-multus-certs\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-etc-openvswitch\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-run-openvswitch\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-run-multus-certs\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-multus-cni-dir\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6aeb9bac-7199-48cc-8159-16017f479f7e-cni-binary-copy\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-etc-openvswitch\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-run-openvswitch\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.669618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dfd2dab-c07b-44bf-b150-7e295cd0794c-ovnkube-config\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dfd2dab-c07b-44bf-b150-7e295cd0794c-ovn-node-metrics-cert\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667636    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667646    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-multus-cni-dir\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n6wq\" (UniqueName: \"kubernetes.io/projected/2dfd2dab-c07b-44bf-b150-7e295cd0794c-kube-api-access-8n6wq\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667695    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/46de41b3-6907-4352-81cf-1216677e98c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667710    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a73e698d-df57-4277-9435-f7d5bfc3be6f-kubelet-config\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-sysctl-d\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667758    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a73e698d-df57-4277-9435-f7d5bfc3be6f-kubelet-config\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-run-k8s-cni-cncf-io\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-run-netns\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-cni-bin\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-modprobe-d\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-systemd\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-sysctl-d\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667904    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-run-netns\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-tuned\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.670394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667938    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-cni-bin\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667942    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-host-run-k8s-cni-cncf-io\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d92t6\" (UniqueName: \"kubernetes.io/projected/3b0fd941-35fa-4dac-a59e-38cbea3386d5-kube-api-access-d92t6\") pod \"network-metrics-daemon-hrrnm\" (UID: \"3b0fd941-35fa-4dac-a59e-38cbea3386d5\") " pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.667997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-systemd\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668027    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-modprobe-d\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668057    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-multus-socket-dir-parent\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-slash\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-log-socket\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dfd2dab-c07b-44bf-b150-7e295cd0794c-ovnkube-script-lib\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668150    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6aeb9bac-7199-48cc-8159-16017f479f7e-cni-binary-copy\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-multus-socket-dir-parent\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-socket-dir\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-log-socket\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668206    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-slash\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668282    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dfd2dab-c07b-44bf-b150-7e295cd0794c-ovnkube-config\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46de41b3-6907-4352-81cf-1216677e98c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-device-dir\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.671270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-device-dir\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-system-cni-dir\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668396    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cac2eb90-87d1-4c96-aede-34bc47a5b958-socket-dir\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb9bac-7199-48cc-8159-16017f479f7e-system-cni-dir\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a73e698d-df57-4277-9435-f7d5bfc3be6f-dbus\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dfd2dab-c07b-44bf-b150-7e295cd0794c-host-run-ovn-kubernetes\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a73e698d-df57-4277-9435-f7d5bfc3be6f-dbus\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac35ebd4-906b-44ad-b7c3-9be83389def6-tmp\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:00.668574    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46de41b3-6907-4352-81cf-1216677e98c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.668600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d867daeb-9f34-4224-91e7-65889fc3e051-host\") pod \"node-ca-zrs62\" (UID: \"d867daeb-9f34-4224-91e7-65889fc3e051\") " pod="openshift-image-registry/node-ca-zrs62"
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.669159    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d867daeb-9f34-4224-91e7-65889fc3e051-host\") pod \"node-ca-zrs62\" (UID: \"d867daeb-9f34-4224-91e7-65889fc3e051\") " pod="openshift-image-registry/node-ca-zrs62"
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:00.669242    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret podName:a73e698d-df57-4277-9435-f7d5bfc3be6f nodeName:}" failed. No retries permitted until 2026-09-16 07:34:01.169210157 +0000 UTC m=+2.095822304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret") pod "global-pull-secret-syncer-xlmf2" (UID: "a73e698d-df57-4277-9435-f7d5bfc3be6f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.669261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46de41b3-6907-4352-81cf-1216677e98c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.669302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dfd2dab-c07b-44bf-b150-7e295cd0794c-ovnkube-script-lib\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.672244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.670920    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dfd2dab-c07b-44bf-b150-7e295cd0794c-ovn-node-metrics-cert\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.673104 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.670931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac35ebd4-906b-44ad-b7c3-9be83389def6-etc-tuned\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.673104 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.671330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac35ebd4-906b-44ad-b7c3-9be83389def6-tmp\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.677305 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.677268    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqc56\" (UniqueName: \"kubernetes.io/projected/cac2eb90-87d1-4c96-aede-34bc47a5b958-kube-api-access-sqc56\") pod \"aws-ebs-csi-driver-node-sbhzq\" (UID: \"cac2eb90-87d1-4c96-aede-34bc47a5b958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.677305 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.677291    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxqcb\" (UniqueName: \"kubernetes.io/projected/a1b6df6f-8ab5-4abb-87b9-8947b5b36043-kube-api-access-sxqcb\") pod \"node-resolver-pw9bz\" (UID: \"a1b6df6f-8ab5-4abb-87b9-8947b5b36043\") " pod="openshift-dns/node-resolver-pw9bz"
Sep 16 07:34:00.677725 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.677696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn7xq\" (UniqueName: \"kubernetes.io/projected/ac35ebd4-906b-44ad-b7c3-9be83389def6-kube-api-access-kn7xq\") pod \"tuned-gsfxp\" (UID: \"ac35ebd4-906b-44ad-b7c3-9be83389def6\") " pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.677899 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.677741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njxj2\" (UniqueName: \"kubernetes.io/projected/6aeb9bac-7199-48cc-8159-16017f479f7e-kube-api-access-njxj2\") pod \"multus-h8wqx\" (UID: \"6aeb9bac-7199-48cc-8159-16017f479f7e\") " pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.679246 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.679226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zqqn\" (UniqueName: \"kubernetes.io/projected/d867daeb-9f34-4224-91e7-65889fc3e051-kube-api-access-2zqqn\") pod \"node-ca-zrs62\" (UID: \"d867daeb-9f34-4224-91e7-65889fc3e051\") " pod="openshift-image-registry/node-ca-zrs62"
Sep 16 07:34:00.679351 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.679262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wtqk\" (UniqueName: \"kubernetes.io/projected/46de41b3-6907-4352-81cf-1216677e98c7-kube-api-access-9wtqk\") pod \"multus-additional-cni-plugins-qvcgx\" (UID: \"46de41b3-6907-4352-81cf-1216677e98c7\") " pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.679642 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.679618    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d92t6\" (UniqueName: \"kubernetes.io/projected/3b0fd941-35fa-4dac-a59e-38cbea3386d5-kube-api-access-d92t6\") pod \"network-metrics-daemon-hrrnm\" (UID: \"3b0fd941-35fa-4dac-a59e-38cbea3386d5\") " pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:00.680077 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.680059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n6wq\" (UniqueName: \"kubernetes.io/projected/2dfd2dab-c07b-44bf-b150-7e295cd0794c-kube-api-access-8n6wq\") pod \"ovnkube-node-gsjsp\" (UID: \"2dfd2dab-c07b-44bf-b150-7e295cd0794c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.702644 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:00.702617    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a7e92e9ab0a02d77b710fd18e6783c.slice/crio-d7affe8a5175a021c3444164c8f9d27f7f74506eb82ae7858e3173d6c7f20657 WatchSource:0}: Error finding container d7affe8a5175a021c3444164c8f9d27f7f74506eb82ae7858e3173d6c7f20657: Status 404 returned error can't find the container with id d7affe8a5175a021c3444164c8f9d27f7f74506eb82ae7858e3173d6c7f20657
Sep 16 07:34:00.702863 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:00.702840    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62629d7452b8a7ebac926f07ec860193.slice/crio-ad841de2d50a8799b78178d3d42803b6b5e6bb5082e640464f19492f50d09188 WatchSource:0}: Error finding container ad841de2d50a8799b78178d3d42803b6b5e6bb5082e640464f19492f50d09188: Status 404 returned error can't find the container with id ad841de2d50a8799b78178d3d42803b6b5e6bb5082e640464f19492f50d09188
Sep 16 07:34:00.709464 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.709448    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 07:34:00.787325 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.787299    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fsz9s"
Sep 16 07:34:00.796352 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:00.796173    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e718e8_9332_436d_b35b_89253c6710a7.slice/crio-0c93cad7037343019ecc2b28333cd6fe0f0279dd3a66ec72ec80213ee369b2f0 WatchSource:0}: Error finding container 0c93cad7037343019ecc2b28333cd6fe0f0279dd3a66ec72ec80213ee369b2f0: Status 404 returned error can't find the container with id 0c93cad7037343019ecc2b28333cd6fe0f0279dd3a66ec72ec80213ee369b2f0
Sep 16 07:34:00.812781 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.812763    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dfg8w"
Sep 16 07:34:00.819645 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:00.819620    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93c2061_f013_4be5_8520_ce0fe0f89c04.slice/crio-6c3201c91cb442c3a9f2d61d9e23de9dabfb80e15d6ec3f125d0a2dc02eb7c88 WatchSource:0}: Error finding container 6c3201c91cb442c3a9f2d61d9e23de9dabfb80e15d6ec3f125d0a2dc02eb7c88: Status 404 returned error can't find the container with id 6c3201c91cb442c3a9f2d61d9e23de9dabfb80e15d6ec3f125d0a2dc02eb7c88
Sep 16 07:34:00.828820 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.828802    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qvcgx"
Sep 16 07:34:00.835918 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:00.835885    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46de41b3_6907_4352_81cf_1216677e98c7.slice/crio-de0f01c310407707d3b1ff139fb5123c556eb9426806eed82324c9bca6387759 WatchSource:0}: Error finding container de0f01c310407707d3b1ff139fb5123c556eb9426806eed82324c9bca6387759: Status 404 returned error can't find the container with id de0f01c310407707d3b1ff139fb5123c556eb9426806eed82324c9bca6387759
Sep 16 07:34:00.842844 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.842828    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h8wqx"
Sep 16 07:34:00.850697 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:00.850680    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aeb9bac_7199_48cc_8159_16017f479f7e.slice/crio-5172d0edb7fca69de1b5941f0d805e5053a0b648bec024550560649a369ad0c9 WatchSource:0}: Error finding container 5172d0edb7fca69de1b5941f0d805e5053a0b648bec024550560649a369ad0c9: Status 404 returned error can't find the container with id 5172d0edb7fca69de1b5941f0d805e5053a0b648bec024550560649a369ad0c9
Sep 16 07:34:00.859523 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.859504    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:00.866116 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:00.866097    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfd2dab_c07b_44bf_b150_7e295cd0794c.slice/crio-0e78fe49111d3d95c5127596d0bb70072d4c5d932c8062bc3226102c4bc55a18 WatchSource:0}: Error finding container 0e78fe49111d3d95c5127596d0bb70072d4c5d932c8062bc3226102c4bc55a18: Status 404 returned error can't find the container with id 0e78fe49111d3d95c5127596d0bb70072d4c5d932c8062bc3226102c4bc55a18
Sep 16 07:34:00.871558 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.871539    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 07:34:00.876917 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.876889    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq"
Sep 16 07:34:00.883559 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:00.883540    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac2eb90_87d1_4c96_aede_34bc47a5b958.slice/crio-e80ee95a8f69bd7fe9c70b75210e220c24c667d642bde8ae3f8346f19db361d0 WatchSource:0}: Error finding container e80ee95a8f69bd7fe9c70b75210e220c24c667d642bde8ae3f8346f19db361d0: Status 404 returned error can't find the container with id e80ee95a8f69bd7fe9c70b75210e220c24c667d642bde8ae3f8346f19db361d0
Sep 16 07:34:00.893879 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.893864    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zrs62"
Sep 16 07:34:00.899673 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.899656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gsfxp"
Sep 16 07:34:00.900523 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:00.900497    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd867daeb_9f34_4224_91e7_65889fc3e051.slice/crio-6493c510c583a3eef8037f062a997b5e25ef4c265acc15a58f6d70e6a31f61e8 WatchSource:0}: Error finding container 6493c510c583a3eef8037f062a997b5e25ef4c265acc15a58f6d70e6a31f61e8: Status 404 returned error can't find the container with id 6493c510c583a3eef8037f062a997b5e25ef4c265acc15a58f6d70e6a31f61e8
Sep 16 07:34:00.908873 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:00.908822    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac35ebd4_906b_44ad_b7c3_9be83389def6.slice/crio-8ac32f1311c8378b21d5d499952ef20b16899b56cc7f16960cf28aa8ac973cf9 WatchSource:0}: Error finding container 8ac32f1311c8378b21d5d499952ef20b16899b56cc7f16960cf28aa8ac973cf9: Status 404 returned error can't find the container with id 8ac32f1311c8378b21d5d499952ef20b16899b56cc7f16960cf28aa8ac973cf9
Sep 16 07:34:00.909910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:00.909880    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pw9bz"
Sep 16 07:34:00.917150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:00.917125    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b6df6f_8ab5_4abb_87b9_8947b5b36043.slice/crio-10bd153aed6279c0e4d5c0655639dba8eaea573326916982b50d30c68cb42272 WatchSource:0}: Error finding container 10bd153aed6279c0e4d5c0655639dba8eaea573326916982b50d30c68cb42272: Status 404 returned error can't find the container with id 10bd153aed6279c0e4d5c0655639dba8eaea573326916982b50d30c68cb42272
Sep 16 07:34:01.174036 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.173952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs\") pod \"network-metrics-daemon-hrrnm\" (UID: \"3b0fd941-35fa-4dac-a59e-38cbea3386d5\") " pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:01.174036 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.174005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:01.174236 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.174071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh5tq\" (UniqueName: \"kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq\") pod \"network-check-target-h4pbq\" (UID: \"8cdad5a8-3714-41e1-8f19-bd579800dc49\") " pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:01.174236 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:01.174108    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 07:34:01.174236 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:01.174160    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret podName:a73e698d-df57-4277-9435-f7d5bfc3be6f nodeName:}" failed. No retries permitted until 2026-09-16 07:34:02.17414671 +0000 UTC m=+3.100758842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret") pod "global-pull-secret-syncer-xlmf2" (UID: "a73e698d-df57-4277-9435-f7d5bfc3be6f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 07:34:01.174236 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:01.174193    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 07:34:01.174236 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:01.174209    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 07:34:01.174236 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:01.174219    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vh5tq for pod openshift-network-diagnostics/network-check-target-h4pbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 07:34:01.174236 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:01.174229    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 07:34:01.174488 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:01.174273    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs podName:3b0fd941-35fa-4dac-a59e-38cbea3386d5 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:02.174259437 +0000 UTC m=+3.100871587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs") pod "network-metrics-daemon-hrrnm" (UID: "3b0fd941-35fa-4dac-a59e-38cbea3386d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 07:34:01.174488 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:01.174292    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq podName:8cdad5a8-3714-41e1-8f19-bd579800dc49 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:02.174280524 +0000 UTC m=+3.100892657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vh5tq" (UniqueName: "kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq") pod "network-check-target-h4pbq" (UID: "8cdad5a8-3714-41e1-8f19-bd579800dc49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 07:34:01.218134 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.218103    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 07:34:01.303120 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.303088    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 07:34:01.532787 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.532676    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 07:29:01 +0000 UTC" deadline="2028-06-20 08:47:43.614256568 +0000 UTC"
Sep 16 07:34:01.532787 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.532727    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15433h13m42.081534641s"
Sep 16 07:34:01.682494 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.680051    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:01.682494 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:01.680180    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:01.716224 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.716118    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal" event={"ID":"c1a7e92e9ab0a02d77b710fd18e6783c","Type":"ContainerStarted","Data":"d7affe8a5175a021c3444164c8f9d27f7f74506eb82ae7858e3173d6c7f20657"}
Sep 16 07:34:01.723355 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.723316    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gsfxp" event={"ID":"ac35ebd4-906b-44ad-b7c3-9be83389def6","Type":"ContainerStarted","Data":"8ac32f1311c8378b21d5d499952ef20b16899b56cc7f16960cf28aa8ac973cf9"}
Sep 16 07:34:01.735440 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.735406    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zrs62" event={"ID":"d867daeb-9f34-4224-91e7-65889fc3e051","Type":"ContainerStarted","Data":"6493c510c583a3eef8037f062a997b5e25ef4c265acc15a58f6d70e6a31f61e8"}
Sep 16 07:34:01.748639 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.748531    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h8wqx" event={"ID":"6aeb9bac-7199-48cc-8159-16017f479f7e","Type":"ContainerStarted","Data":"5172d0edb7fca69de1b5941f0d805e5053a0b648bec024550560649a369ad0c9"}
Sep 16 07:34:01.757300 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.757263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvcgx" event={"ID":"46de41b3-6907-4352-81cf-1216677e98c7","Type":"ContainerStarted","Data":"de0f01c310407707d3b1ff139fb5123c556eb9426806eed82324c9bca6387759"}
Sep 16 07:34:01.767841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.767807    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dfg8w" event={"ID":"d93c2061-f013-4be5-8520-ce0fe0f89c04","Type":"ContainerStarted","Data":"6c3201c91cb442c3a9f2d61d9e23de9dabfb80e15d6ec3f125d0a2dc02eb7c88"}
Sep 16 07:34:01.795168 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.795092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fsz9s" event={"ID":"76e718e8-9332-436d-b35b-89253c6710a7","Type":"ContainerStarted","Data":"0c93cad7037343019ecc2b28333cd6fe0f0279dd3a66ec72ec80213ee369b2f0"}
Sep 16 07:34:01.800129 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.800100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal" event={"ID":"62629d7452b8a7ebac926f07ec860193","Type":"ContainerStarted","Data":"ad841de2d50a8799b78178d3d42803b6b5e6bb5082e640464f19492f50d09188"}
Sep 16 07:34:01.807463 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.807412    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pw9bz" event={"ID":"a1b6df6f-8ab5-4abb-87b9-8947b5b36043","Type":"ContainerStarted","Data":"10bd153aed6279c0e4d5c0655639dba8eaea573326916982b50d30c68cb42272"}
Sep 16 07:34:01.822279 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.822248    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq" event={"ID":"cac2eb90-87d1-4c96-aede-34bc47a5b958","Type":"ContainerStarted","Data":"e80ee95a8f69bd7fe9c70b75210e220c24c667d642bde8ae3f8346f19db361d0"}
Sep 16 07:34:01.839579 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:01.839546    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp" event={"ID":"2dfd2dab-c07b-44bf-b150-7e295cd0794c","Type":"ContainerStarted","Data":"0e78fe49111d3d95c5127596d0bb70072d4c5d932c8062bc3226102c4bc55a18"}
Sep 16 07:34:02.194209 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:02.194116    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh5tq\" (UniqueName: \"kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq\") pod \"network-check-target-h4pbq\" (UID: \"8cdad5a8-3714-41e1-8f19-bd579800dc49\") " pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:02.194209 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:02.194182    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs\") pod \"network-metrics-daemon-hrrnm\" (UID: \"3b0fd941-35fa-4dac-a59e-38cbea3386d5\") " pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:02.194449 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:02.194239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:02.194449 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:02.194374    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 07:34:02.194449 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:02.194436    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret podName:a73e698d-df57-4277-9435-f7d5bfc3be6f nodeName:}" failed. No retries permitted until 2026-09-16 07:34:04.194417356 +0000 UTC m=+5.121029498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret") pod "global-pull-secret-syncer-xlmf2" (UID: "a73e698d-df57-4277-9435-f7d5bfc3be6f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 07:34:02.194918 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:02.194880    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 07:34:02.194918 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:02.194916    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 07:34:02.195061 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:02.194930    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vh5tq for pod openshift-network-diagnostics/network-check-target-h4pbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 07:34:02.195061 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:02.194977    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq podName:8cdad5a8-3714-41e1-8f19-bd579800dc49 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:04.194962201 +0000 UTC m=+5.121574334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vh5tq" (UniqueName: "kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq") pod "network-check-target-h4pbq" (UID: "8cdad5a8-3714-41e1-8f19-bd579800dc49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 07:34:02.195061 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:02.195045    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 07:34:02.195228 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:02.195087    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs podName:3b0fd941-35fa-4dac-a59e-38cbea3386d5 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:04.195072516 +0000 UTC m=+5.121684656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs") pod "network-metrics-daemon-hrrnm" (UID: "3b0fd941-35fa-4dac-a59e-38cbea3386d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 07:34:02.533671 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:02.533548    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 07:29:01 +0000 UTC" deadline="2028-04-10 04:54:54.332806107 +0000 UTC"
Sep 16 07:34:02.533671 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:02.533620    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13725h20m51.799190378s"
Sep 16 07:34:02.678870 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:02.678155    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:02.678870 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:02.678289    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:02.678870 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:02.678710    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:02.678870 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:02.678823    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:03.295996 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:03.295965    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 07:34:03.680778 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:03.680695    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:03.681222 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:03.680925    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:04.212007 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:04.211972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:04.212204 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:04.212038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh5tq\" (UniqueName: \"kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq\") pod \"network-check-target-h4pbq\" (UID: \"8cdad5a8-3714-41e1-8f19-bd579800dc49\") " pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:04.212204 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:04.212071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs\") pod \"network-metrics-daemon-hrrnm\" (UID: \"3b0fd941-35fa-4dac-a59e-38cbea3386d5\") " pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:04.212323 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:04.212225    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 07:34:04.212323 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:04.212277    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 07:34:04.212420 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:04.212291    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs podName:3b0fd941-35fa-4dac-a59e-38cbea3386d5 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:08.212270253 +0000 UTC m=+9.138882402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs") pod "network-metrics-daemon-hrrnm" (UID: "3b0fd941-35fa-4dac-a59e-38cbea3386d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 07:34:04.212420 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:04.212334    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 07:34:04.212420 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:04.212356    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 07:34:04.212420 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:04.212362    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret podName:a73e698d-df57-4277-9435-f7d5bfc3be6f nodeName:}" failed. No retries permitted until 2026-09-16 07:34:08.212343533 +0000 UTC m=+9.138955671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret") pod "global-pull-secret-syncer-xlmf2" (UID: "a73e698d-df57-4277-9435-f7d5bfc3be6f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 07:34:04.212420 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:04.212370    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vh5tq for pod openshift-network-diagnostics/network-check-target-h4pbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 07:34:04.212420 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:04.212421    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq podName:8cdad5a8-3714-41e1-8f19-bd579800dc49 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:08.212409769 +0000 UTC m=+9.139021903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vh5tq" (UniqueName: "kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq") pod "network-check-target-h4pbq" (UID: "8cdad5a8-3714-41e1-8f19-bd579800dc49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 07:34:04.678375 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:04.677705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:04.678375 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:04.677731    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:04.678375 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:04.677874    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:04.678375 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:04.678331    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:05.677772 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:05.677735    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:05.678408 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:05.677909    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:06.678749 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:06.677985    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:06.678749 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:06.678133    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:06.678749 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:06.678595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:06.678749 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:06.678707    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:07.678484 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:07.678450    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:07.678786 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:07.678579    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:08.248269 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:08.247760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:08.248269 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:08.247818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh5tq\" (UniqueName: \"kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq\") pod \"network-check-target-h4pbq\" (UID: \"8cdad5a8-3714-41e1-8f19-bd579800dc49\") " pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:08.248269 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:08.247874    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs\") pod \"network-metrics-daemon-hrrnm\" (UID: \"3b0fd941-35fa-4dac-a59e-38cbea3386d5\") " pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:08.248269 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:08.248002    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 07:34:08.248269 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:08.248063    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs podName:3b0fd941-35fa-4dac-a59e-38cbea3386d5 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:16.24804203 +0000 UTC m=+17.174654165 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs") pod "network-metrics-daemon-hrrnm" (UID: "3b0fd941-35fa-4dac-a59e-38cbea3386d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 07:34:08.248567 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:08.248414    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 07:34:08.248567 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:08.248440    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 07:34:08.248567 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:08.248452    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vh5tq for pod openshift-network-diagnostics/network-check-target-h4pbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 07:34:08.248567 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:08.248495    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 07:34:08.248567 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:08.248508    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq podName:8cdad5a8-3714-41e1-8f19-bd579800dc49 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:16.24849139 +0000 UTC m=+17.175103524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vh5tq" (UniqueName: "kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq") pod "network-check-target-h4pbq" (UID: "8cdad5a8-3714-41e1-8f19-bd579800dc49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 07:34:08.248567 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:08.248542    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret podName:a73e698d-df57-4277-9435-f7d5bfc3be6f nodeName:}" failed. No retries permitted until 2026-09-16 07:34:16.248530898 +0000 UTC m=+17.175143032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret") pod "global-pull-secret-syncer-xlmf2" (UID: "a73e698d-df57-4277-9435-f7d5bfc3be6f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 07:34:08.678332 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:08.678255    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:08.678509 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:08.678386    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:08.678509 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:08.678450    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:08.678638 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:08.678582    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:09.678805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:09.678770    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:09.679341 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:09.678879    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:10.678430 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:10.677804    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:10.678430 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:10.677952    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:10.678430 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:10.678302    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:10.678430 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:10.678388    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:11.677912 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:11.677877    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:11.678454 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:11.678008    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:12.678293 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:12.678261    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:12.678680 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:12.678261    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:12.678680 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:12.678359    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:12.678680 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:12.678482    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:13.678106 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:13.678026    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:13.678251 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:13.678156    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:14.677451 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:14.677420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:14.677920 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:14.677420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:14.677920 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:14.677542    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:14.677920 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:14.677673    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:15.678154 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:15.678127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:15.678546 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:15.678231    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:16.307695 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:16.307654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:16.307914 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:16.307705    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh5tq\" (UniqueName: \"kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq\") pod \"network-check-target-h4pbq\" (UID: \"8cdad5a8-3714-41e1-8f19-bd579800dc49\") " pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:16.307914 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:16.307737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs\") pod \"network-metrics-daemon-hrrnm\" (UID: \"3b0fd941-35fa-4dac-a59e-38cbea3386d5\") " pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:16.307914 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:16.307803    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 07:34:16.307914 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:16.307889    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret podName:a73e698d-df57-4277-9435-f7d5bfc3be6f nodeName:}" failed. No retries permitted until 2026-09-16 07:34:32.307862375 +0000 UTC m=+33.234474507 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret") pod "global-pull-secret-syncer-xlmf2" (UID: "a73e698d-df57-4277-9435-f7d5bfc3be6f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 07:34:16.308114 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:16.307925    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 07:34:16.308114 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:16.307937    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 07:34:16.308114 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:16.307949    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 07:34:16.308114 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:16.307964    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vh5tq for pod openshift-network-diagnostics/network-check-target-h4pbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 07:34:16.308114 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:16.307998    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs podName:3b0fd941-35fa-4dac-a59e-38cbea3386d5 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:32.307977592 +0000 UTC m=+33.234589737 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs") pod "network-metrics-daemon-hrrnm" (UID: "3b0fd941-35fa-4dac-a59e-38cbea3386d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 07:34:16.308114 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:16.308019    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq podName:8cdad5a8-3714-41e1-8f19-bd579800dc49 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:32.308008481 +0000 UTC m=+33.234620612 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vh5tq" (UniqueName: "kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq") pod "network-check-target-h4pbq" (UID: "8cdad5a8-3714-41e1-8f19-bd579800dc49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 07:34:16.677605 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:16.677535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:16.677768 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:16.677542    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:16.677768 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:16.677631    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:16.677768 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:16.677717    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:17.677941 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:17.677905    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:17.678396 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:17.678043    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:18.678024 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:18.677990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:18.678484 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:18.678124    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:18.678484 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:18.678179    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:18.678484 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:18.678286    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:19.678450 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:19.678420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:19.678876 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:19.678524    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:20.678288 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:20.678265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:20.678368 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:20.678266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:20.678368 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:20.678357    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:20.678448 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:20.678436    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:20.890165 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:20.890138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp" event={"ID":"2dfd2dab-c07b-44bf-b150-7e295cd0794c","Type":"ContainerStarted","Data":"94bac1cad48e799ae0a137705f28dcbf3440e067ec7572ae7604b6871d58a1e3"}
Sep 16 07:34:20.892516 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:20.892147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal" event={"ID":"c1a7e92e9ab0a02d77b710fd18e6783c","Type":"ContainerStarted","Data":"7a496c6794eb4b17047e336da49d5fe268eec74ed56e9f02941db044c7a13575"}
Sep 16 07:34:20.896249 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:20.896224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gsfxp" event={"ID":"ac35ebd4-906b-44ad-b7c3-9be83389def6","Type":"ContainerStarted","Data":"4cf881705451aea591ab436dea6d71d909e1bfda6d9ca9f78e1ecc631d6e369a"}
Sep 16 07:34:20.924539 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:20.924499    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal" podStartSLOduration=19.924485831 podStartE2EDuration="19.924485831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:20.906730821 +0000 UTC m=+21.833342974" watchObservedRunningTime="2026-09-16 07:34:20.924485831 +0000 UTC m=+21.851097992"
Sep 16 07:34:20.924630 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:20.924608    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gsfxp" podStartSLOduration=1.252124477 podStartE2EDuration="20.924602739s" totalImagesPullingTime="19.672478262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:20.924275457 +0000 UTC m=+21.850887621" watchObservedRunningTime="2026-09-16 07:34:20.924602739 +0000 UTC m=+21.851214891"
Sep 16 07:34:21.677947 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.677703    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:21.678102 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:21.678047    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:21.900211 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.900132    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zrs62" event={"ID":"d867daeb-9f34-4224-91e7-65889fc3e051","Type":"ContainerStarted","Data":"0e45ed90fc2138052aa40cce7bdeb0a438f833b6dbda26bb80603d6570b28401"}
Sep 16 07:34:21.901939 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.901914    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h8wqx" event={"ID":"6aeb9bac-7199-48cc-8159-16017f479f7e","Type":"ContainerStarted","Data":"fb72aa0843fcd5fb78e714e10308b668815dff30e6b1e066fc16ff48a8c61757"}
Sep 16 07:34:21.903779 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.903751    2581 generic.go:356] "Generic (PLEG): container finished" podID="46de41b3-6907-4352-81cf-1216677e98c7" containerID="790606ee06902d2d84419523ac38efedd803fd291ab3e2d13a41101fc1820005" exitCode=0
Sep 16 07:34:21.903924 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.903849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvcgx" event={"ID":"46de41b3-6907-4352-81cf-1216677e98c7","Type":"ContainerDied","Data":"790606ee06902d2d84419523ac38efedd803fd291ab3e2d13a41101fc1820005"}
Sep 16 07:34:21.905990 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.905967    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fsz9s" event={"ID":"76e718e8-9332-436d-b35b-89253c6710a7","Type":"ContainerStarted","Data":"f85c6fc6d1a5c326c35f3405edc8bdd0bdf103e289041689e8eebe3383b62fbd"}
Sep 16 07:34:21.908124 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.908102    2581 generic.go:356] "Generic (PLEG): container finished" podID="62629d7452b8a7ebac926f07ec860193" containerID="459479fbcf0f609656210f1c74015d349a6b9362f25bee809e742b4d79c735c7" exitCode=0
Sep 16 07:34:21.908279 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.908188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal" event={"ID":"62629d7452b8a7ebac926f07ec860193","Type":"ContainerDied","Data":"459479fbcf0f609656210f1c74015d349a6b9362f25bee809e742b4d79c735c7"}
Sep 16 07:34:21.909981 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.909957    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pw9bz" event={"ID":"a1b6df6f-8ab5-4abb-87b9-8947b5b36043","Type":"ContainerStarted","Data":"ab627d620bff743ca8e1b41434559ee1b810685c0d8d3d752c6e53ae8419d761"}
Sep 16 07:34:21.911808 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.911785    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq" event={"ID":"cac2eb90-87d1-4c96-aede-34bc47a5b958","Type":"ContainerStarted","Data":"974f706198e859a737acc186a3847a8e8205bd333958b86c775928175f5f354b"}
Sep 16 07:34:21.914708 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.914685    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 07:34:21.914891 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.914850    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zrs62" podStartSLOduration=2.255394169 podStartE2EDuration="21.914833567s" totalImagesPullingTime="19.659439398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:21.914554968 +0000 UTC m=+22.841167122" watchObservedRunningTime="2026-09-16 07:34:21.914833567 +0000 UTC m=+22.841445717"
Sep 16 07:34:21.915068 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.915051    2581 generic.go:356] "Generic (PLEG): container finished" podID="2dfd2dab-c07b-44bf-b150-7e295cd0794c" containerID="1f6956baa9afa083dea123d0806b9e5a80d74dfe5317b103be94db00eb7413a8" exitCode=1
Sep 16 07:34:21.915149 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.915125    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp" event={"ID":"2dfd2dab-c07b-44bf-b150-7e295cd0794c","Type":"ContainerStarted","Data":"324e653dfafd95ef9d6e408be565445f8efa19591926b20c8dd20555a9b39998"}
Sep 16 07:34:21.915218 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.915203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp" event={"ID":"2dfd2dab-c07b-44bf-b150-7e295cd0794c","Type":"ContainerStarted","Data":"ab3b463459d1c51907546101b67823bb0d89c86016aa7a61082af3f526054061"}
Sep 16 07:34:21.915273 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.915252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp" event={"ID":"2dfd2dab-c07b-44bf-b150-7e295cd0794c","Type":"ContainerStarted","Data":"8496eeffbcfeaa91633314e2a6a62da0d1f127c35b188857105452d389cd8e96"}
Sep 16 07:34:21.915326 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.915301    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp" event={"ID":"2dfd2dab-c07b-44bf-b150-7e295cd0794c","Type":"ContainerStarted","Data":"b6e85d61e5982d5670a888a7bf8cc173d8c577094dffa73095c8c9393df55ba2"}
Sep 16 07:34:21.915379 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.915349    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp" event={"ID":"2dfd2dab-c07b-44bf-b150-7e295cd0794c","Type":"ContainerDied","Data":"1f6956baa9afa083dea123d0806b9e5a80d74dfe5317b103be94db00eb7413a8"}
Sep 16 07:34:21.933204 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.933172    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h8wqx" podStartSLOduration=1.584704963 podStartE2EDuration="21.933160936s" totalImagesPullingTime="20.348455973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:21.932598105 +0000 UTC m=+22.859210271" watchObservedRunningTime="2026-09-16 07:34:21.933160936 +0000 UTC m=+22.859773088"
Sep 16 07:34:21.947933 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.947899    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pw9bz" podStartSLOduration=2.28037691 podStartE2EDuration="21.947886427s" totalImagesPullingTime="19.667509517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:21.947692786 +0000 UTC m=+22.874304942" watchObservedRunningTime="2026-09-16 07:34:21.947886427 +0000 UTC m=+22.874498570"
Sep 16 07:34:21.977132 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:21.977095    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fsz9s" podStartSLOduration=2.211831504 podStartE2EDuration="21.977083295s" totalImagesPullingTime="19.765251791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:21.976339042 +0000 UTC m=+22.902951208" watchObservedRunningTime="2026-09-16 07:34:21.977083295 +0000 UTC m=+22.903695450"
Sep 16 07:34:22.614174 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:22.614152    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 07:34:22.657772 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:22.657688    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T07:34:22.614168991Z","UUID":"bb82e320-3150-49b9-a838-67afda20453c","Handler":null,"Name":"","Endpoint":""}
Sep 16 07:34:22.660197 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:22.659997    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 07:34:22.660197 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:22.660035    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 07:34:22.677318 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:22.677293    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:22.677387 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:22.677325    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:22.677429 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:22.677400    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:22.677535 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:22.677517    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:22.919225 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:22.919127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal" event={"ID":"62629d7452b8a7ebac926f07ec860193","Type":"ContainerStarted","Data":"1a6e67ba9a5406480ba6f84078e02feaace1321e82684bad1011112da85448ad"}
Sep 16 07:34:22.920911 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:22.920882    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq" event={"ID":"cac2eb90-87d1-4c96-aede-34bc47a5b958","Type":"ContainerStarted","Data":"d8bf7de13c4a61362dfeca8e9e381e85ab1ebf059740d54f22a4d473ebc4b858"}
Sep 16 07:34:22.922310 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:22.922283    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dfg8w" event={"ID":"d93c2061-f013-4be5-8520-ce0fe0f89c04","Type":"ContainerStarted","Data":"f6a218811cec3b09c94056b5da1dfe8820278e7f83f9df60816027d75826d9f5"}
Sep 16 07:34:22.935240 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:22.935203    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal" podStartSLOduration=21.935192183 podStartE2EDuration="21.935192183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:22.934967317 +0000 UTC m=+23.861579472" watchObservedRunningTime="2026-09-16 07:34:22.935192183 +0000 UTC m=+23.861804336"
Sep 16 07:34:22.949549 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:22.949511    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dfg8w" podStartSLOduration=3.20783938 podStartE2EDuration="22.949500472s" totalImagesPullingTime="19.741661092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:22.949279797 +0000 UTC m=+23.875891949" watchObservedRunningTime="2026-09-16 07:34:22.949500472 +0000 UTC m=+23.876112625"
Sep 16 07:34:23.678324 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:23.678256    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:23.678457 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:23.678380    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:23.926583 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:23.926540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq" event={"ID":"cac2eb90-87d1-4c96-aede-34bc47a5b958","Type":"ContainerStarted","Data":"d6659ad8b47a917b5ca4adb04f6cda9a825b3f8fbc17794cde12616ab2089afd"}
Sep 16 07:34:23.930338 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:23.930277    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 07:34:23.930759 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:23.930734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp" event={"ID":"2dfd2dab-c07b-44bf-b150-7e295cd0794c","Type":"ContainerStarted","Data":"cb6fd682044e892434690e1d01a483e3070c2c0de95ad887fa9029fcc919c96d"}
Sep 16 07:34:23.945918 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:23.945875    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbhzq" podStartSLOduration=2.067763045 podStartE2EDuration="23.945861978s" totalImagesPullingTime="21.878098933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:34:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:23.945746027 +0000 UTC m=+24.872358182" watchObservedRunningTime="2026-09-16 07:34:23.945861978 +0000 UTC m=+24.872474131"
Sep 16 07:34:24.677435 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:24.677409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:24.677615 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:24.677408    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:24.677615 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:24.677556    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:24.677615 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:24.677592    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:25.348683 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:25.348654    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fsz9s"
Sep 16 07:34:25.349450 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:25.349428    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fsz9s"
Sep 16 07:34:25.677988 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:25.677945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:25.678142 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:25.678081    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:25.939086 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:25.938868    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 07:34:25.939538 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:25.939441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp" event={"ID":"2dfd2dab-c07b-44bf-b150-7e295cd0794c","Type":"ContainerStarted","Data":"cb6129fced748ef19ad69f30eebb1e9fa60ff3ddf06707c9a4a930f76d208dc5"}
Sep 16 07:34:25.939790 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:25.939765    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fsz9s"
Sep 16 07:34:25.940158 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:25.940135    2581 scope.go:117] "RemoveContainer" containerID="1f6956baa9afa083dea123d0806b9e5a80d74dfe5317b103be94db00eb7413a8"
Sep 16 07:34:25.940351 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:25.940327    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fsz9s"
Sep 16 07:34:26.678249 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:26.678218    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:26.678712 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:26.678350    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:26.678712 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:26.678381    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:26.678712 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:26.678450    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:27.449299 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:27.449270    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4pbq"]
Sep 16 07:34:27.449432 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:27.449371    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:27.449500 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:27.449476    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:27.452111 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:27.452085    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xlmf2"]
Sep 16 07:34:27.452214 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:27.452179    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:27.452291 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:27.452275    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:27.452721 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:27.452702    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hrrnm"]
Sep 16 07:34:27.452802 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:27.452774    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:27.452892 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:27.452877    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:27.946093 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:27.946020    2581 generic.go:356] "Generic (PLEG): container finished" podID="46de41b3-6907-4352-81cf-1216677e98c7" containerID="2d649cee4b86a026f0fd6a2f84a487b2419a9ca555ba3f7fe1826d6b262ef58c" exitCode=0
Sep 16 07:34:27.946937 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:27.946112    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvcgx" event={"ID":"46de41b3-6907-4352-81cf-1216677e98c7","Type":"ContainerDied","Data":"2d649cee4b86a026f0fd6a2f84a487b2419a9ca555ba3f7fe1826d6b262ef58c"}
Sep 16 07:34:27.949870 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:27.949854    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 07:34:27.950231 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:27.950211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp" event={"ID":"2dfd2dab-c07b-44bf-b150-7e295cd0794c","Type":"ContainerStarted","Data":"00d3fb5c91d330dce563046b20b7352733f2622fbc6da6c533ecf4580e683720"}
Sep 16 07:34:27.950418 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:27.950398    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:27.950486 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:27.950470    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:27.950651 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:27.950622    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:27.965476 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:27.965456    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:27.969187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:27.969172    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:34:28.004904 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:28.004863    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp" podStartSLOduration=8.275536259 podStartE2EDuration="28.004848706s" totalImagesPullingTime="19.729312447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:28.00356827 +0000 UTC m=+28.930180423" watchObservedRunningTime="2026-09-16 07:34:28.004848706 +0000 UTC m=+28.931460858"
Sep 16 07:34:28.677665 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:28.677492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:28.677860 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:28.677494    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:28.677860 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:28.677765    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:28.677860 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:28.677807    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:29.680851 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:29.680807    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:29.681264 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:29.681005    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:29.956526 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:29.956486    2581 generic.go:356] "Generic (PLEG): container finished" podID="46de41b3-6907-4352-81cf-1216677e98c7" containerID="599afc7f552512bd42c88887e8103d7a76371bf0c23cf862b822ebb20085ef0a" exitCode=0
Sep 16 07:34:29.956694 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:29.956550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvcgx" event={"ID":"46de41b3-6907-4352-81cf-1216677e98c7","Type":"ContainerDied","Data":"599afc7f552512bd42c88887e8103d7a76371bf0c23cf862b822ebb20085ef0a"}
Sep 16 07:34:30.677670 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:30.677638    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:30.677847 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:30.677689    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:30.677847 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:30.677773    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:30.677929 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:30.677893    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:30.961964 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:30.961930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvcgx" event={"ID":"46de41b3-6907-4352-81cf-1216677e98c7","Type":"ContainerStarted","Data":"969aebd51996da241c366ee525e7d97a2552959369ddba64fa9fb2eae5d79a9e"}
Sep 16 07:34:31.677581 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:31.677552    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:31.677757 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:31.677672    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:31.969109 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:31.969034    2581 generic.go:356] "Generic (PLEG): container finished" podID="46de41b3-6907-4352-81cf-1216677e98c7" containerID="969aebd51996da241c366ee525e7d97a2552959369ddba64fa9fb2eae5d79a9e" exitCode=0
Sep 16 07:34:31.969109 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:31.969093    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvcgx" event={"ID":"46de41b3-6907-4352-81cf-1216677e98c7","Type":"ContainerDied","Data":"969aebd51996da241c366ee525e7d97a2552959369ddba64fa9fb2eae5d79a9e"}
Sep 16 07:34:32.326883 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:32.326738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh5tq\" (UniqueName: \"kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq\") pod \"network-check-target-h4pbq\" (UID: \"8cdad5a8-3714-41e1-8f19-bd579800dc49\") " pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:32.326883 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:32.326786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs\") pod \"network-metrics-daemon-hrrnm\" (UID: \"3b0fd941-35fa-4dac-a59e-38cbea3386d5\") " pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:32.326883 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:32.326873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:32.327150 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:32.326935    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 07:34:32.327150 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:32.326956    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 07:34:32.327150 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:32.326963    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 07:34:32.327150 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:32.326970    2581 projected.go:196] Error preparing data for projected volume kube-api-access-vh5tq for pod openshift-network-diagnostics/network-check-target-h4pbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 07:34:32.327150 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:32.327023    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret podName:a73e698d-df57-4277-9435-f7d5bfc3be6f nodeName:}" failed. No retries permitted until 2026-09-16 07:35:04.327005333 +0000 UTC m=+65.253617467 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret") pod "global-pull-secret-syncer-xlmf2" (UID: "a73e698d-df57-4277-9435-f7d5bfc3be6f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 07:34:32.327150 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:32.327040    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq podName:8cdad5a8-3714-41e1-8f19-bd579800dc49 nodeName:}" failed. No retries permitted until 2026-09-16 07:35:04.327031642 +0000 UTC m=+65.253643773 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vh5tq" (UniqueName: "kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq") pod "network-check-target-h4pbq" (UID: "8cdad5a8-3714-41e1-8f19-bd579800dc49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 07:34:32.327150 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:32.327103    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 07:34:32.327150 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:32.327135    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs podName:3b0fd941-35fa-4dac-a59e-38cbea3386d5 nodeName:}" failed. No retries permitted until 2026-09-16 07:35:04.327124412 +0000 UTC m=+65.253736547 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs") pod "network-metrics-daemon-hrrnm" (UID: "3b0fd941-35fa-4dac-a59e-38cbea3386d5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 07:34:32.678223 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:32.678188    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:32.678395 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:32.678231    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:32.678395 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:32.678311    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4pbq" podUID="8cdad5a8-3714-41e1-8f19-bd579800dc49"
Sep 16 07:34:32.678524 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:32.678409    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlmf2" podUID="a73e698d-df57-4277-9435-f7d5bfc3be6f"
Sep 16 07:34:33.677715 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.677688    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:33.678044 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:33.677789    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrnm" podUID="3b0fd941-35fa-4dac-a59e-38cbea3386d5"
Sep 16 07:34:33.820420 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.820392    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pw9bz_a1b6df6f-8ab5-4abb-87b9-8947b5b36043/dns-node-resolver/0.log"
Sep 16 07:34:33.850442 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.850378    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeReady"
Sep 16 07:34:33.850573 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.850519    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 07:34:33.891928 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.891891    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jltjj"]
Sep 16 07:34:33.895287 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.895268    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gqj9q"]
Sep 16 07:34:33.895396 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.895380    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:33.897431 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.897407    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 07:34:33.897431 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.897411    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 07:34:33.897431 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.897430    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2msf4\""
Sep 16 07:34:33.899067 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.899049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:33.901149 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.901125    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 07:34:33.901149 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.901143    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 07:34:33.901268 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.901248    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 07:34:33.901575 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.901558    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 07:34:33.901924 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.901907    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-57x9m\""
Sep 16 07:34:33.903237 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.903149    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jltjj"]
Sep 16 07:34:33.909474 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.909451    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gqj9q"]
Sep 16 07:34:33.911931 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.911913    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vcffj"]
Sep 16 07:34:33.915409 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.915393    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vcffj"
Sep 16 07:34:33.918356 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.918334    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 07:34:33.918448 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.918429    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 07:34:33.918540 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.918337    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 07:34:33.919174 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.919124    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mh2l7\""
Sep 16 07:34:33.924350 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:33.924330    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vcffj"]
Sep 16 07:34:34.038594 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.038564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kjwj\" (UniqueName: \"kubernetes.io/projected/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-kube-api-access-6kjwj\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:34.038735 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.038612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-tmp-dir\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:34.038735 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.038648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:34.038735 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.038667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c32b025-8552-45d9-90ca-df22eadccbab-data-volume\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:34.038735 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.038685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmgh\" (UniqueName: \"kubernetes.io/projected/2c32b025-8552-45d9-90ca-df22eadccbab-kube-api-access-jvmgh\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:34.038735 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.038701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/216fb3f5-f1f8-4739-80bf-8d0254b57a8f-cert\") pod \"ingress-canary-vcffj\" (UID: \"216fb3f5-f1f8-4739-80bf-8d0254b57a8f\") " pod="openshift-ingress-canary/ingress-canary-vcffj"
Sep 16 07:34:34.038735 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.038716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c32b025-8552-45d9-90ca-df22eadccbab-crio-socket\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:34.038993 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.038755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c32b025-8552-45d9-90ca-df22eadccbab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:34.038993 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.038787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:34.038993 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.038807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjhls\" (UniqueName: \"kubernetes.io/projected/216fb3f5-f1f8-4739-80bf-8d0254b57a8f-kube-api-access-jjhls\") pod \"ingress-canary-vcffj\" (UID: \"216fb3f5-f1f8-4739-80bf-8d0254b57a8f\") " pod="openshift-ingress-canary/ingress-canary-vcffj"
Sep 16 07:34:34.038993 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.038898    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-config-volume\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:34.139908 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.139833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-config-volume\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:34.140036 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.139912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kjwj\" (UniqueName: \"kubernetes.io/projected/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-kube-api-access-6kjwj\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:34.140036 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.139958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-tmp-dir\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:34.140036 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.139984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:34.140036 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.140016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c32b025-8552-45d9-90ca-df22eadccbab-data-volume\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:34.140235 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.140043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmgh\" (UniqueName: \"kubernetes.io/projected/2c32b025-8552-45d9-90ca-df22eadccbab-kube-api-access-jvmgh\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:34.140235 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:34.140128    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 07:34:34.140235 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:34.140197    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls podName:2c32b025-8552-45d9-90ca-df22eadccbab nodeName:}" failed. No retries permitted until 2026-09-16 07:34:34.640175782 +0000 UTC m=+35.566787930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gqj9q" (UID: "2c32b025-8552-45d9-90ca-df22eadccbab") : secret "insights-runtime-extractor-tls" not found
Sep 16 07:34:34.140235 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.140221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/216fb3f5-f1f8-4739-80bf-8d0254b57a8f-cert\") pod \"ingress-canary-vcffj\" (UID: \"216fb3f5-f1f8-4739-80bf-8d0254b57a8f\") " pod="openshift-ingress-canary/ingress-canary-vcffj"
Sep 16 07:34:34.140440 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.140257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c32b025-8552-45d9-90ca-df22eadccbab-crio-socket\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:34.140440 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.140284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c32b025-8552-45d9-90ca-df22eadccbab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:34.140440 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.140320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:34.140440 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.140324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-tmp-dir\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:34.140440 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.140343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjhls\" (UniqueName: \"kubernetes.io/projected/216fb3f5-f1f8-4739-80bf-8d0254b57a8f-kube-api-access-jjhls\") pod \"ingress-canary-vcffj\" (UID: \"216fb3f5-f1f8-4739-80bf-8d0254b57a8f\") " pod="openshift-ingress-canary/ingress-canary-vcffj"
Sep 16 07:34:34.140440 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.140410    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c32b025-8552-45d9-90ca-df22eadccbab-data-volume\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:34.140702 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:34.140497    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 07:34:34.140702 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.140498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-config-volume\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:34.140702 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:34.140551    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 07:34:34.140702 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:34.140570    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c32b025-8552-45d9-90ca-df22eadccbab-kube-rbac-proxy-cm podName:2c32b025-8552-45d9-90ca-df22eadccbab nodeName:}" failed. No retries permitted until 2026-09-16 07:34:34.640552059 +0000 UTC m=+35.567164197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c32b025-8552-45d9-90ca-df22eadccbab-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gqj9q" (UID: "2c32b025-8552-45d9-90ca-df22eadccbab") : configmap "kube-rbac-proxy" not found
Sep 16 07:34:34.140702 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:34.140587    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls podName:fc0d14f2-f58e-40ee-9110-8ce6a97ea069 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:34.640578442 +0000 UTC m=+35.567190573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls") pod "dns-default-jltjj" (UID: "fc0d14f2-f58e-40ee-9110-8ce6a97ea069") : secret "dns-default-metrics-tls" not found
Sep 16 07:34:34.140702 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.140612    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c32b025-8552-45d9-90ca-df22eadccbab-crio-socket\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:34.145607 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.145445    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/216fb3f5-f1f8-4739-80bf-8d0254b57a8f-cert\") pod \"ingress-canary-vcffj\" (UID: \"216fb3f5-f1f8-4739-80bf-8d0254b57a8f\") " pod="openshift-ingress-canary/ingress-canary-vcffj"
Sep 16 07:34:34.151890 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.151863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kjwj\" (UniqueName: \"kubernetes.io/projected/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-kube-api-access-6kjwj\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:34.152474 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.152432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmgh\" (UniqueName: \"kubernetes.io/projected/2c32b025-8552-45d9-90ca-df22eadccbab-kube-api-access-jvmgh\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:34.152908 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.152888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjhls\" (UniqueName: \"kubernetes.io/projected/216fb3f5-f1f8-4739-80bf-8d0254b57a8f-kube-api-access-jjhls\") pod \"ingress-canary-vcffj\" (UID: \"216fb3f5-f1f8-4739-80bf-8d0254b57a8f\") " pod="openshift-ingress-canary/ingress-canary-vcffj"
Sep 16 07:34:34.225742 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.225709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vcffj"
Sep 16 07:34:34.509725 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.509694    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vcffj"]
Sep 16 07:34:34.514409 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:34.514377    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216fb3f5_f1f8_4739_80bf_8d0254b57a8f.slice/crio-f97f43b1cb1223316604d59442fcdc96f5f0365ec5eb961a3dd2231f1d6a126e WatchSource:0}: Error finding container f97f43b1cb1223316604d59442fcdc96f5f0365ec5eb961a3dd2231f1d6a126e: Status 404 returned error can't find the container with id f97f43b1cb1223316604d59442fcdc96f5f0365ec5eb961a3dd2231f1d6a126e
Sep 16 07:34:34.643053 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.643018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:34.643196 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.643063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c32b025-8552-45d9-90ca-df22eadccbab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:34.643196 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.643086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:34.643196 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:34.643164    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 07:34:34.643196 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:34.643185    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 07:34:34.643196 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:34.643198    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 07:34:34.643405 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:34.643240    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c32b025-8552-45d9-90ca-df22eadccbab-kube-rbac-proxy-cm podName:2c32b025-8552-45d9-90ca-df22eadccbab nodeName:}" failed. No retries permitted until 2026-09-16 07:34:35.643219902 +0000 UTC m=+36.569832046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c32b025-8552-45d9-90ca-df22eadccbab-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gqj9q" (UID: "2c32b025-8552-45d9-90ca-df22eadccbab") : configmap "kube-rbac-proxy" not found
Sep 16 07:34:34.643405 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:34.643259    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls podName:fc0d14f2-f58e-40ee-9110-8ce6a97ea069 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:35.643250377 +0000 UTC m=+36.569862516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls") pod "dns-default-jltjj" (UID: "fc0d14f2-f58e-40ee-9110-8ce6a97ea069") : secret "dns-default-metrics-tls" not found
Sep 16 07:34:34.643405 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:34.643274    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls podName:2c32b025-8552-45d9-90ca-df22eadccbab nodeName:}" failed. No retries permitted until 2026-09-16 07:34:35.643265981 +0000 UTC m=+36.569878117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gqj9q" (UID: "2c32b025-8552-45d9-90ca-df22eadccbab") : secret "insights-runtime-extractor-tls" not found
Sep 16 07:34:34.677446 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.677418    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:34:34.677559 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.677419    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:34:34.679951 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.679929    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 07:34:34.680422 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.680024    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 07:34:34.680422 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.680331    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 07:34:34.680422 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.680382    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lz8hv\""
Sep 16 07:34:34.979046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:34.979002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vcffj" event={"ID":"216fb3f5-f1f8-4739-80bf-8d0254b57a8f","Type":"ContainerStarted","Data":"f97f43b1cb1223316604d59442fcdc96f5f0365ec5eb961a3dd2231f1d6a126e"}
Sep 16 07:34:35.004428 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.004402    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zrs62_d867daeb-9f34-4224-91e7-65889fc3e051/node-ca/0.log"
Sep 16 07:34:35.650615 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.650581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:35.650800 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.650639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c32b025-8552-45d9-90ca-df22eadccbab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:35.650800 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.650675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:35.650800 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:35.650744    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 07:34:35.650800 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:35.650790    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 07:34:35.651043 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:35.650836    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls podName:2c32b025-8552-45d9-90ca-df22eadccbab nodeName:}" failed. No retries permitted until 2026-09-16 07:34:37.650801827 +0000 UTC m=+38.577413981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gqj9q" (UID: "2c32b025-8552-45d9-90ca-df22eadccbab") : secret "insights-runtime-extractor-tls" not found
Sep 16 07:34:35.651043 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:35.650865    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c32b025-8552-45d9-90ca-df22eadccbab-kube-rbac-proxy-cm podName:2c32b025-8552-45d9-90ca-df22eadccbab nodeName:}" failed. No retries permitted until 2026-09-16 07:34:37.650847897 +0000 UTC m=+38.577460052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c32b025-8552-45d9-90ca-df22eadccbab-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gqj9q" (UID: "2c32b025-8552-45d9-90ca-df22eadccbab") : configmap "kube-rbac-proxy" not found
Sep 16 07:34:35.651043 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:35.650797    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 07:34:35.651043 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:35.650906    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls podName:fc0d14f2-f58e-40ee-9110-8ce6a97ea069 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:37.650893949 +0000 UTC m=+38.577506113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls") pod "dns-default-jltjj" (UID: "fc0d14f2-f58e-40ee-9110-8ce6a97ea069") : secret "dns-default-metrics-tls" not found
Sep 16 07:34:35.677990 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.677961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:34:35.680391 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.680370    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 07:34:35.680391 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.680387    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zf7hx\""
Sep 16 07:34:35.791648 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.791617    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-29pbg"]
Sep 16 07:34:35.795296 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.795268    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-29pbg"
Sep 16 07:34:35.797809 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.797703    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 07:34:35.797809 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.797726    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 07:34:35.798520 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.798503    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gwws7\""
Sep 16 07:34:35.798687 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.798663    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 07:34:35.798794 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.798731    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 07:34:35.805715 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.805693    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-29pbg"]
Sep 16 07:34:35.953659 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.953623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcgm9\" (UniqueName: \"kubernetes.io/projected/7c52cbd3-dede-45b4-9fa8-6858e55fe824-kube-api-access-jcgm9\") pod \"service-ca-568869fdc5-29pbg\" (UID: \"7c52cbd3-dede-45b4-9fa8-6858e55fe824\") " pod="openshift-service-ca/service-ca-568869fdc5-29pbg"
Sep 16 07:34:35.953848 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.953724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7c52cbd3-dede-45b4-9fa8-6858e55fe824-signing-cabundle\") pod \"service-ca-568869fdc5-29pbg\" (UID: \"7c52cbd3-dede-45b4-9fa8-6858e55fe824\") " pod="openshift-service-ca/service-ca-568869fdc5-29pbg"
Sep 16 07:34:35.953848 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:35.953775    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7c52cbd3-dede-45b4-9fa8-6858e55fe824-signing-key\") pod \"service-ca-568869fdc5-29pbg\" (UID: \"7c52cbd3-dede-45b4-9fa8-6858e55fe824\") " pod="openshift-service-ca/service-ca-568869fdc5-29pbg"
Sep 16 07:34:36.054341 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:36.054305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcgm9\" (UniqueName: \"kubernetes.io/projected/7c52cbd3-dede-45b4-9fa8-6858e55fe824-kube-api-access-jcgm9\") pod \"service-ca-568869fdc5-29pbg\" (UID: \"7c52cbd3-dede-45b4-9fa8-6858e55fe824\") " pod="openshift-service-ca/service-ca-568869fdc5-29pbg"
Sep 16 07:34:36.054506 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:36.054396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7c52cbd3-dede-45b4-9fa8-6858e55fe824-signing-cabundle\") pod \"service-ca-568869fdc5-29pbg\" (UID: \"7c52cbd3-dede-45b4-9fa8-6858e55fe824\") " pod="openshift-service-ca/service-ca-568869fdc5-29pbg"
Sep 16 07:34:36.054506 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:36.054440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7c52cbd3-dede-45b4-9fa8-6858e55fe824-signing-key\") pod \"service-ca-568869fdc5-29pbg\" (UID: \"7c52cbd3-dede-45b4-9fa8-6858e55fe824\") " pod="openshift-service-ca/service-ca-568869fdc5-29pbg"
Sep 16 07:34:36.055885 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:36.055859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7c52cbd3-dede-45b4-9fa8-6858e55fe824-signing-cabundle\") pod \"service-ca-568869fdc5-29pbg\" (UID: \"7c52cbd3-dede-45b4-9fa8-6858e55fe824\") " pod="openshift-service-ca/service-ca-568869fdc5-29pbg"
Sep 16 07:34:36.058035 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:36.058014    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7c52cbd3-dede-45b4-9fa8-6858e55fe824-signing-key\") pod \"service-ca-568869fdc5-29pbg\" (UID: \"7c52cbd3-dede-45b4-9fa8-6858e55fe824\") " pod="openshift-service-ca/service-ca-568869fdc5-29pbg"
Sep 16 07:34:36.062779 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:36.062756    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcgm9\" (UniqueName: \"kubernetes.io/projected/7c52cbd3-dede-45b4-9fa8-6858e55fe824-kube-api-access-jcgm9\") pod \"service-ca-568869fdc5-29pbg\" (UID: \"7c52cbd3-dede-45b4-9fa8-6858e55fe824\") " pod="openshift-service-ca/service-ca-568869fdc5-29pbg"
Sep 16 07:34:36.107871 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:36.107845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-29pbg"
Sep 16 07:34:37.624648 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:37.624530    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-29pbg"]
Sep 16 07:34:37.628230 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:37.628202    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c52cbd3_dede_45b4_9fa8_6858e55fe824.slice/crio-cd30b94fbd81fd71bb26c8ada2664bfce0e9f79569a882bbc9f7b4cb1389139c WatchSource:0}: Error finding container cd30b94fbd81fd71bb26c8ada2664bfce0e9f79569a882bbc9f7b4cb1389139c: Status 404 returned error can't find the container with id cd30b94fbd81fd71bb26c8ada2664bfce0e9f79569a882bbc9f7b4cb1389139c
Sep 16 07:34:37.667494 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:37.667469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c32b025-8552-45d9-90ca-df22eadccbab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:37.667575 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:37.667508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:37.667621 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:37.667609    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 07:34:37.667665 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:37.667658    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls podName:fc0d14f2-f58e-40ee-9110-8ce6a97ea069 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:41.667644221 +0000 UTC m=+42.594256352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls") pod "dns-default-jltjj" (UID: "fc0d14f2-f58e-40ee-9110-8ce6a97ea069") : secret "dns-default-metrics-tls" not found
Sep 16 07:34:37.667716 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:37.667691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:37.667765 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:37.667754    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 07:34:37.667810 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:37.667780    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls podName:2c32b025-8552-45d9-90ca-df22eadccbab nodeName:}" failed. No retries permitted until 2026-09-16 07:34:41.667773593 +0000 UTC m=+42.594385724 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gqj9q" (UID: "2c32b025-8552-45d9-90ca-df22eadccbab") : secret "insights-runtime-extractor-tls" not found
Sep 16 07:34:37.668010 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:37.667992    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c32b025-8552-45d9-90ca-df22eadccbab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:37.987789 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:37.987753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-29pbg" event={"ID":"7c52cbd3-dede-45b4-9fa8-6858e55fe824","Type":"ContainerStarted","Data":"cd30b94fbd81fd71bb26c8ada2664bfce0e9f79569a882bbc9f7b4cb1389139c"}
Sep 16 07:34:37.989237 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:37.989209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vcffj" event={"ID":"216fb3f5-f1f8-4739-80bf-8d0254b57a8f","Type":"ContainerStarted","Data":"b751df851822604809ebb3c47be32180a8660a13ca6bc34f46c93af7893df0ef"}
Sep 16 07:34:37.991804 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:37.991781    2581 generic.go:356] "Generic (PLEG): container finished" podID="46de41b3-6907-4352-81cf-1216677e98c7" containerID="c92758111e673831129a87d24de6259d3a25272e86026d43eb2259a11a0802a6" exitCode=0
Sep 16 07:34:37.991929 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:37.991812    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvcgx" event={"ID":"46de41b3-6907-4352-81cf-1216677e98c7","Type":"ContainerDied","Data":"c92758111e673831129a87d24de6259d3a25272e86026d43eb2259a11a0802a6"}
Sep 16 07:34:38.006949 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:38.006907    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vcffj" podStartSLOduration=2.154512965 podStartE2EDuration="5.006889585s" totalImagesPullingTime="2.85237662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:38.00614472 +0000 UTC m=+38.932756873" watchObservedRunningTime="2026-09-16 07:34:38.006889585 +0000 UTC m=+38.933501738"
Sep 16 07:34:38.997976 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:38.997942    2581 generic.go:356] "Generic (PLEG): container finished" podID="46de41b3-6907-4352-81cf-1216677e98c7" containerID="509f1ee278e1004f74d9d758ae192667fd5cec593b21768e264462f97dac163f" exitCode=0
Sep 16 07:34:38.998467 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:38.998031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvcgx" event={"ID":"46de41b3-6907-4352-81cf-1216677e98c7","Type":"ContainerDied","Data":"509f1ee278e1004f74d9d758ae192667fd5cec593b21768e264462f97dac163f"}
Sep 16 07:34:40.003586 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:40.003553    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvcgx" event={"ID":"46de41b3-6907-4352-81cf-1216677e98c7","Type":"ContainerStarted","Data":"a53859c02e5473902a701284e62234e99d476de85e6a3f153741af1bcd894aa1"}
Sep 16 07:34:41.007665 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:41.007623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-29pbg" event={"ID":"7c52cbd3-dede-45b4-9fa8-6858e55fe824","Type":"ContainerStarted","Data":"7b857e0c822ec2ce63852425b47401a96daef1dc5ed7509465f895a6af5163e5"}
Sep 16 07:34:41.023738 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:41.023682    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qvcgx" podStartSLOduration=11.937104206 podStartE2EDuration="41.023663081s" totalImagesPullingTime="29.086558875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:34:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:40.028495796 +0000 UTC m=+40.955107952" watchObservedRunningTime="2026-09-16 07:34:41.023663081 +0000 UTC m=+41.950275226"
Sep 16 07:34:41.023934 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:41.023901    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-29pbg" podStartSLOduration=3.3053842429999998 podStartE2EDuration="6.023892684s" totalImagesPullingTime="2.718508441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:41.022570733 +0000 UTC m=+41.949182887" watchObservedRunningTime="2026-09-16 07:34:41.023892684 +0000 UTC m=+41.950504838"
Sep 16 07:34:41.699125 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:41.699091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:41.699307 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:41.699148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:41.699307 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:41.699264    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 07:34:41.699307 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:41.699270    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 07:34:41.699480 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:41.699331    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls podName:fc0d14f2-f58e-40ee-9110-8ce6a97ea069 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:49.699312167 +0000 UTC m=+50.625924299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls") pod "dns-default-jltjj" (UID: "fc0d14f2-f58e-40ee-9110-8ce6a97ea069") : secret "dns-default-metrics-tls" not found
Sep 16 07:34:41.699480 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:34:41.699350    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls podName:2c32b025-8552-45d9-90ca-df22eadccbab nodeName:}" failed. No retries permitted until 2026-09-16 07:34:49.699341905 +0000 UTC m=+50.625954039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gqj9q" (UID: "2c32b025-8552-45d9-90ca-df22eadccbab") : secret "insights-runtime-extractor-tls" not found
Sep 16 07:34:49.750064 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:49.750026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:49.750455 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:49.750151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:49.753114 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:49.753085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c32b025-8552-45d9-90ca-df22eadccbab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gqj9q\" (UID: \"2c32b025-8552-45d9-90ca-df22eadccbab\") " pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:49.753114 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:49.753099    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc0d14f2-f58e-40ee-9110-8ce6a97ea069-metrics-tls\") pod \"dns-default-jltjj\" (UID: \"fc0d14f2-f58e-40ee-9110-8ce6a97ea069\") " pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:49.807180 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:49.807152    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:49.811238 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:49.811220    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gqj9q"
Sep 16 07:34:50.109405 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:50.109334    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jltjj"]
Sep 16 07:34:50.112945 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:50.112915    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0d14f2_f58e_40ee_9110_8ce6a97ea069.slice/crio-aa2e18fcf123af28db9bdaa2696573ada68fde038b7cce2909c9132adde2bc9b WatchSource:0}: Error finding container aa2e18fcf123af28db9bdaa2696573ada68fde038b7cce2909c9132adde2bc9b: Status 404 returned error can't find the container with id aa2e18fcf123af28db9bdaa2696573ada68fde038b7cce2909c9132adde2bc9b
Sep 16 07:34:50.138554 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:50.138532    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gqj9q"]
Sep 16 07:34:50.141368 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:50.141343    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c32b025_8552_45d9_90ca_df22eadccbab.slice/crio-6cdf30964280e1929d4987b207c3ee845efeeaa28a98303e88b3a185d9889dac WatchSource:0}: Error finding container 6cdf30964280e1929d4987b207c3ee845efeeaa28a98303e88b3a185d9889dac: Status 404 returned error can't find the container with id 6cdf30964280e1929d4987b207c3ee845efeeaa28a98303e88b3a185d9889dac
Sep 16 07:34:51.034386 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:51.034347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gqj9q" event={"ID":"2c32b025-8552-45d9-90ca-df22eadccbab","Type":"ContainerStarted","Data":"d5bc315ef7178545242a3de33d744af0378c2f4987286788cab5861263163a60"}
Sep 16 07:34:51.034773 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:51.034413    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gqj9q" event={"ID":"2c32b025-8552-45d9-90ca-df22eadccbab","Type":"ContainerStarted","Data":"6cdf30964280e1929d4987b207c3ee845efeeaa28a98303e88b3a185d9889dac"}
Sep 16 07:34:51.035530 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:51.035511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jltjj" event={"ID":"fc0d14f2-f58e-40ee-9110-8ce6a97ea069","Type":"ContainerStarted","Data":"aa2e18fcf123af28db9bdaa2696573ada68fde038b7cce2909c9132adde2bc9b"}
Sep 16 07:34:52.041923 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:52.041886    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gqj9q" event={"ID":"2c32b025-8552-45d9-90ca-df22eadccbab","Type":"ContainerStarted","Data":"073788dc86b124a2416105d531fdd3605cc4f6f2cfb8e9b4c724ca5f68287886"}
Sep 16 07:34:53.053212 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:53.053123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gqj9q" event={"ID":"2c32b025-8552-45d9-90ca-df22eadccbab","Type":"ContainerStarted","Data":"d34c39f235d72fc6257a3bf4bb8f09c0e07a8c64d572564c282923708e98ddfb"}
Sep 16 07:34:53.056081 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:53.056050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jltjj" event={"ID":"fc0d14f2-f58e-40ee-9110-8ce6a97ea069","Type":"ContainerStarted","Data":"405fea800d164cdfbd64ca2c9b9e8e774e9260aa16e7a9283c46708087e73343"}
Sep 16 07:34:53.068999 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:53.068948    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gqj9q" podStartSLOduration=17.48439318 podStartE2EDuration="20.068932154s" totalImagesPullingTime="2.584538974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:34:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:53.068685033 +0000 UTC m=+53.995297185" watchObservedRunningTime="2026-09-16 07:34:53.068932154 +0000 UTC m=+53.995544301"
Sep 16 07:34:54.064173 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:54.064138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jltjj" event={"ID":"fc0d14f2-f58e-40ee-9110-8ce6a97ea069","Type":"ContainerStarted","Data":"6101a98180cf50a091311cb89c63a7b9a827ec7edd576ccebb0803dfe9877be8"}
Sep 16 07:34:54.081802 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:54.081752    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jltjj" podStartSLOduration=18.2625392 podStartE2EDuration="21.081736418s" totalImagesPullingTime="2.819197218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:54.080922299 +0000 UTC m=+55.007534454" watchObservedRunningTime="2026-09-16 07:34:54.081736418 +0000 UTC m=+55.008348568"
Sep 16 07:34:55.067625 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:55.067592    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jltjj"
Sep 16 07:34:56.340174 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.340135    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc"]
Sep 16 07:34:56.341883 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.341864    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"]
Sep 16 07:34:56.342018 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.341998    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc"
Sep 16 07:34:56.343674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.343647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.344392 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.344365    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 07:34:56.344505 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.344472    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 07:34:56.344505 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.344482    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 07:34:56.345382 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.345359    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 07:34:56.345382 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.345379    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-m422v\""
Sep 16 07:34:56.346059 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.346033    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 07:34:56.346355 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.346336    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 07:34:56.346474 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.346355    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 07:34:56.346474 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.346363    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 07:34:56.354416 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.354396    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc"]
Sep 16 07:34:56.357952 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.357930    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"]
Sep 16 07:34:56.393725 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.393703    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.393859 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.393733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8kxj\" (UniqueName: \"kubernetes.io/projected/f81b8a6c-24b3-4315-986a-16ea5d2eb349-kube-api-access-t8kxj\") pod \"managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc\" (UID: \"f81b8a6c-24b3-4315-986a-16ea5d2eb349\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc"
Sep 16 07:34:56.393859 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.393754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.393859 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.393783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-hub\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.393957 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.393861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-ca\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.393957 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.393892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.393957 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.393927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f81b8a6c-24b3-4315-986a-16ea5d2eb349-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc\" (UID: \"f81b8a6c-24b3-4315-986a-16ea5d2eb349\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc"
Sep 16 07:34:56.394046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.393958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42zcc\" (UniqueName: \"kubernetes.io/projected/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-kube-api-access-42zcc\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.494543 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.494521    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.494649 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.494548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8kxj\" (UniqueName: \"kubernetes.io/projected/f81b8a6c-24b3-4315-986a-16ea5d2eb349-kube-api-access-t8kxj\") pod \"managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc\" (UID: \"f81b8a6c-24b3-4315-986a-16ea5d2eb349\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc"
Sep 16 07:34:56.494649 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.494566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.494649 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.494603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-hub\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.494760 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.494739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-ca\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.494792 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.494774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.494873 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.494803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f81b8a6c-24b3-4315-986a-16ea5d2eb349-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc\" (UID: \"f81b8a6c-24b3-4315-986a-16ea5d2eb349\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc"
Sep 16 07:34:56.494873 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.494862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42zcc\" (UniqueName: \"kubernetes.io/projected/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-kube-api-access-42zcc\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.495639 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.495611    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.497226 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.497199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.497303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.497233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-ca\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.497346 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.497320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-hub\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.497510 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.497494    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.497612 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.497598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f81b8a6c-24b3-4315-986a-16ea5d2eb349-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc\" (UID: \"f81b8a6c-24b3-4315-986a-16ea5d2eb349\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc"
Sep 16 07:34:56.507344 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.507324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8kxj\" (UniqueName: \"kubernetes.io/projected/f81b8a6c-24b3-4315-986a-16ea5d2eb349-kube-api-access-t8kxj\") pod \"managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc\" (UID: \"f81b8a6c-24b3-4315-986a-16ea5d2eb349\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc"
Sep 16 07:34:56.508076 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.508060    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42zcc\" (UniqueName: \"kubernetes.io/projected/ddcc6a0f-5254-4d5a-b15a-5412c89bf93a-kube-api-access-42zcc\") pod \"cluster-proxy-proxy-agent-8448959bbb-pt8ws\" (UID: \"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.661081 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.661057    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc"
Sep 16 07:34:56.666322 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.666305    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"
Sep 16 07:34:56.930434 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.930411    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc"]
Sep 16 07:34:56.932948 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:56.932924    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws"]
Sep 16 07:34:56.933267 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:56.933239    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf81b8a6c_24b3_4315_986a_16ea5d2eb349.slice/crio-cf08e0d97a749eedcee7a2b548176fbd4af71f56b09c751875aca188d48ced82 WatchSource:0}: Error finding container cf08e0d97a749eedcee7a2b548176fbd4af71f56b09c751875aca188d48ced82: Status 404 returned error can't find the container with id cf08e0d97a749eedcee7a2b548176fbd4af71f56b09c751875aca188d48ced82
Sep 16 07:34:56.935539 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:34:56.935513    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddcc6a0f_5254_4d5a_b15a_5412c89bf93a.slice/crio-54d6a0e552dbc2977cc8326b63acdfdb28dd8b8f446d8b1083e06d891c6b7b63 WatchSource:0}: Error finding container 54d6a0e552dbc2977cc8326b63acdfdb28dd8b8f446d8b1083e06d891c6b7b63: Status 404 returned error can't find the container with id 54d6a0e552dbc2977cc8326b63acdfdb28dd8b8f446d8b1083e06d891c6b7b63
Sep 16 07:34:57.075781 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:57.075748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc" event={"ID":"f81b8a6c-24b3-4315-986a-16ea5d2eb349","Type":"ContainerStarted","Data":"cf08e0d97a749eedcee7a2b548176fbd4af71f56b09c751875aca188d48ced82"}
Sep 16 07:34:57.076798 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:57.076778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws" event={"ID":"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a","Type":"ContainerStarted","Data":"54d6a0e552dbc2977cc8326b63acdfdb28dd8b8f446d8b1083e06d891c6b7b63"}
Sep 16 07:34:59.970368 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:34:59.970341    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gsjsp"
Sep 16 07:35:00.089891 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:00.089855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc" event={"ID":"f81b8a6c-24b3-4315-986a-16ea5d2eb349","Type":"ContainerStarted","Data":"1fe9e91d41c5a8db76179f0b7b6d99ef0fafcc51299a9da5912af838ff57163b"}
Sep 16 07:35:00.114401 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:00.114348    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66f8fc46f8-cdgfc" podStartSLOduration=1.109239374 podStartE2EDuration="4.114333541s" totalImagesPullingTime="3.005094167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:00.113534357 +0000 UTC m=+61.040146509" watchObservedRunningTime="2026-09-16 07:35:00.114333541 +0000 UTC m=+61.040945693"
Sep 16 07:35:02.098936 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:02.098901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws" event={"ID":"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a","Type":"ContainerStarted","Data":"51add4ebd491af755bed174b30691a17c75fe8a4039df451257160a26a0fda1e"}
Sep 16 07:35:02.098936 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:02.098941    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws" event={"ID":"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a","Type":"ContainerStarted","Data":"87dd8f8324a301a5ebe77a21b395b2c078e80ca296e7416acf43ed6b454adb35"}
Sep 16 07:35:02.099426 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:02.098953    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws" event={"ID":"ddcc6a0f-5254-4d5a-b15a-5412c89bf93a","Type":"ContainerStarted","Data":"07680a0d4f54284445cdf494cb698ab42b9ad674d1f999f469218ceaca54c337"}
Sep 16 07:35:02.117164 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:02.117118    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8448959bbb-pt8ws" podStartSLOduration=1.902570924 podStartE2EDuration="6.117103997s" totalImagesPullingTime="4.214533073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:02.115973826 +0000 UTC m=+63.042585991" watchObservedRunningTime="2026-09-16 07:35:02.117103997 +0000 UTC m=+63.043716151"
Sep 16 07:35:04.351968 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.351928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs\") pod \"network-metrics-daemon-hrrnm\" (UID: \"3b0fd941-35fa-4dac-a59e-38cbea3386d5\") " pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:35:04.352388 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.351999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:35:04.352388 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.352035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh5tq\" (UniqueName: \"kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq\") pod \"network-check-target-h4pbq\" (UID: \"8cdad5a8-3714-41e1-8f19-bd579800dc49\") " pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:35:04.354520 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.354498    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 07:35:04.355420 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.355406    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 07:35:04.355469 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.355416    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 07:35:04.364320 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.364299    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 07:35:04.364587 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.364564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a73e698d-df57-4277-9435-f7d5bfc3be6f-original-pull-secret\") pod \"global-pull-secret-syncer-xlmf2\" (UID: \"a73e698d-df57-4277-9435-f7d5bfc3be6f\") " pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:35:04.364655 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.364638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0fd941-35fa-4dac-a59e-38cbea3386d5-metrics-certs\") pod \"network-metrics-daemon-hrrnm\" (UID: \"3b0fd941-35fa-4dac-a59e-38cbea3386d5\") " pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:35:04.375161 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.375143    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh5tq\" (UniqueName: \"kubernetes.io/projected/8cdad5a8-3714-41e1-8f19-bd579800dc49-kube-api-access-vh5tq\") pod \"network-check-target-h4pbq\" (UID: \"8cdad5a8-3714-41e1-8f19-bd579800dc49\") " pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:35:04.389020 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.389003    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlmf2"
Sep 16 07:35:04.397175 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.397152    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lz8hv\""
Sep 16 07:35:04.405125 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.405101    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:35:04.491805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.491778    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zf7hx\""
Sep 16 07:35:04.499664 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.499643    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrnm"
Sep 16 07:35:04.657327 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.657190    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xlmf2"]
Sep 16 07:35:04.659814 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:35:04.659783    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73e698d_df57_4277_9435_f7d5bfc3be6f.slice/crio-6407f45bdb55a291d92cf5f06b8c1735ef1b344f9487a0e444cd7d3ee6dd6286 WatchSource:0}: Error finding container 6407f45bdb55a291d92cf5f06b8c1735ef1b344f9487a0e444cd7d3ee6dd6286: Status 404 returned error can't find the container with id 6407f45bdb55a291d92cf5f06b8c1735ef1b344f9487a0e444cd7d3ee6dd6286
Sep 16 07:35:04.680320 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.680296    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4pbq"]
Sep 16 07:35:04.682812 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:35:04.682773    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdad5a8_3714_41e1_8f19_bd579800dc49.slice/crio-4a421eb6e9f232a58c57d86c1ffbeb42460cc36a8295e6c5a4477c48e0641ae5 WatchSource:0}: Error finding container 4a421eb6e9f232a58c57d86c1ffbeb42460cc36a8295e6c5a4477c48e0641ae5: Status 404 returned error can't find the container with id 4a421eb6e9f232a58c57d86c1ffbeb42460cc36a8295e6c5a4477c48e0641ae5
Sep 16 07:35:04.763413 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:04.763383    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hrrnm"]
Sep 16 07:35:04.766308 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:35:04.766279    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0fd941_35fa_4dac_a59e_38cbea3386d5.slice/crio-5876fbd335c2e21b66637887571831cd94fc7bb8729ec89108c5575a8c5f5998 WatchSource:0}: Error finding container 5876fbd335c2e21b66637887571831cd94fc7bb8729ec89108c5575a8c5f5998: Status 404 returned error can't find the container with id 5876fbd335c2e21b66637887571831cd94fc7bb8729ec89108c5575a8c5f5998
Sep 16 07:35:05.073466 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:05.073384    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jltjj"
Sep 16 07:35:05.108595 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:05.108565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hrrnm" event={"ID":"3b0fd941-35fa-4dac-a59e-38cbea3386d5","Type":"ContainerStarted","Data":"5876fbd335c2e21b66637887571831cd94fc7bb8729ec89108c5575a8c5f5998"}
Sep 16 07:35:05.109737 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:05.109710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4pbq" event={"ID":"8cdad5a8-3714-41e1-8f19-bd579800dc49","Type":"ContainerStarted","Data":"4a421eb6e9f232a58c57d86c1ffbeb42460cc36a8295e6c5a4477c48e0641ae5"}
Sep 16 07:35:05.111059 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:05.111031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xlmf2" event={"ID":"a73e698d-df57-4277-9435-f7d5bfc3be6f","Type":"ContainerStarted","Data":"6407f45bdb55a291d92cf5f06b8c1735ef1b344f9487a0e444cd7d3ee6dd6286"}
Sep 16 07:35:06.117303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:06.117222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hrrnm" event={"ID":"3b0fd941-35fa-4dac-a59e-38cbea3386d5","Type":"ContainerStarted","Data":"3aafc5e5cc723cfbcb6c91bae2a02f90142024713785e6bb8a3f8d429ec96635"}
Sep 16 07:35:06.117303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:06.117288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hrrnm" event={"ID":"3b0fd941-35fa-4dac-a59e-38cbea3386d5","Type":"ContainerStarted","Data":"4943c3444d4bb026d6f419f5c9f0e2b7630f8c4f737af0b00aab53b0d2a15026"}
Sep 16 07:35:06.132721 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:06.132672    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hrrnm" podStartSLOduration=65.090870896 podStartE2EDuration="1m6.132655609s" totalImagesPullingTime="1.041784713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:06.132176676 +0000 UTC m=+67.058788831" watchObservedRunningTime="2026-09-16 07:35:06.132655609 +0000 UTC m=+67.059267762"
Sep 16 07:35:10.138659 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.138625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xlmf2" event={"ID":"a73e698d-df57-4277-9435-f7d5bfc3be6f","Type":"ContainerStarted","Data":"4a306c1a1d869ad04f711c2051d442d0c4bdc23a82fc2df981dea4634c66b755"}
Sep 16 07:35:10.140218 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.140187    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4pbq" event={"ID":"8cdad5a8-3714-41e1-8f19-bd579800dc49","Type":"ContainerStarted","Data":"a1f38faf8db3d9adfc2465645ed67e49f127723f3db7e372f27093066b518a4e"}
Sep 16 07:35:10.140348 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.140326    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:35:10.152507 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.152458    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xlmf2" podStartSLOduration=65.254954696 podStartE2EDuration="1m10.152446001s" totalImagesPullingTime="4.897491305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:10.152098001 +0000 UTC m=+71.078710165" watchObservedRunningTime="2026-09-16 07:35:10.152446001 +0000 UTC m=+71.079058154"
Sep 16 07:35:10.165804 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.165766    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h4pbq" podStartSLOduration=65.295329353 podStartE2EDuration="1m10.165753947s" totalImagesPullingTime="4.870424594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:10.165447455 +0000 UTC m=+71.092059608" watchObservedRunningTime="2026-09-16 07:35:10.165753947 +0000 UTC m=+71.092366100"
Sep 16 07:35:10.663411 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.663373    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tb859"]
Sep 16 07:35:10.685473 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.685446    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.687949 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.687927    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kqtql\""
Sep 16 07:35:10.688100 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.687942    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 07:35:10.688164 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.688002    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 07:35:10.688217 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.688027    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 07:35:10.688972 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.688951    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 07:35:10.689056 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.689019    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 07:35:10.689154 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.689129    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 07:35:10.803842 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.803805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c906e2cd-06e6-4c74-a699-096dcc52f2b1-metrics-client-ca\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.803979 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.803846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c906e2cd-06e6-4c74-a699-096dcc52f2b1-sys\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.803979 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.803868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-accelerators-collector-config\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.803979 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.803886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-textfile\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.803979 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.803910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.803979 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.803934    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c906e2cd-06e6-4c74-a699-096dcc52f2b1-root\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.803979 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.803956    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-tls\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.804191 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.804039    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9lpx\" (UniqueName: \"kubernetes.io/projected/c906e2cd-06e6-4c74-a699-096dcc52f2b1-kube-api-access-b9lpx\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.804191 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.804082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-wtmp\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.905038 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.905014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c906e2cd-06e6-4c74-a699-096dcc52f2b1-root\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.905201 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.905049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-tls\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.905201 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.905075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9lpx\" (UniqueName: \"kubernetes.io/projected/c906e2cd-06e6-4c74-a699-096dcc52f2b1-kube-api-access-b9lpx\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.905201 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.905101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-wtmp\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.905201 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.905125    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c906e2cd-06e6-4c74-a699-096dcc52f2b1-root\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.905201 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:35:10.905182    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 07:35:10.905461 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.905133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c906e2cd-06e6-4c74-a699-096dcc52f2b1-metrics-client-ca\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.905461 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:35:10.905246    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-tls podName:c906e2cd-06e6-4c74-a699-096dcc52f2b1 nodeName:}" failed. No retries permitted until 2026-09-16 07:35:11.405224801 +0000 UTC m=+72.331836955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-tls") pod "node-exporter-tb859" (UID: "c906e2cd-06e6-4c74-a699-096dcc52f2b1") : secret "node-exporter-tls" not found
Sep 16 07:35:10.905461 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.905268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c906e2cd-06e6-4c74-a699-096dcc52f2b1-sys\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.905461 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.905241    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-wtmp\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.905461 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.905303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-accelerators-collector-config\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.905461 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.905333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-textfile\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.905461 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.905337    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c906e2cd-06e6-4c74-a699-096dcc52f2b1-sys\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.905461 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.905430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.905770 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.905590    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-textfile\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.905770 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.905698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c906e2cd-06e6-4c74-a699-096dcc52f2b1-metrics-client-ca\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.905770 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.905757    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-accelerators-collector-config\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.907545 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.907524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:10.918887 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:10.918813    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9lpx\" (UniqueName: \"kubernetes.io/projected/c906e2cd-06e6-4c74-a699-096dcc52f2b1-kube-api-access-b9lpx\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:11.409950 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.409922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-tls\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:11.412163 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.412141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c906e2cd-06e6-4c74-a699-096dcc52f2b1-node-exporter-tls\") pod \"node-exporter-tb859\" (UID: \"c906e2cd-06e6-4c74-a699-096dcc52f2b1\") " pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:11.595627 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.595594    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tb859"
Sep 16 07:35:11.634182 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:35:11.634153    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc906e2cd_06e6_4c74_a699_096dcc52f2b1.slice/crio-cff0a9809f543aea178d3be2879efe6ae3f4741cbe806f748376b955bc669b7f WatchSource:0}: Error finding container cff0a9809f543aea178d3be2879efe6ae3f4741cbe806f748376b955bc669b7f: Status 404 returned error can't find the container with id cff0a9809f543aea178d3be2879efe6ae3f4741cbe806f748376b955bc669b7f
Sep 16 07:35:11.697465 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.697383    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 07:35:11.727107 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.727071    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 07:35:11.727251 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.727229    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.730022 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.729997    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 07:35:11.730171 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.730032    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-s8rxs\""
Sep 16 07:35:11.730171 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.730084    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 07:35:11.730171 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.730099    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 07:35:11.730171 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.730114    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 07:35:11.730171 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.730036    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 07:35:11.730441 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.730418    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 07:35:11.730441 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.730422    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 07:35:11.730540 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.730461    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 07:35:11.730540 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.730507    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 07:35:11.812715 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.812688    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.812715 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.812717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.812918 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.812732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/74a7f0bd-9010-456c-a708-e0625f4e2ea2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.812918 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.812750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-web-config\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.812918 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.812769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74a7f0bd-9010-456c-a708-e0625f4e2ea2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.812918 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.812785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.812918 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.812854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74a7f0bd-9010-456c-a708-e0625f4e2ea2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.812918 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.812880    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hntv\" (UniqueName: \"kubernetes.io/projected/74a7f0bd-9010-456c-a708-e0625f4e2ea2-kube-api-access-7hntv\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.812918 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.812897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.813124 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.812922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-config-volume\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.813124 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.812987    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.813124 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.813017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74a7f0bd-9010-456c-a708-e0625f4e2ea2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.813124 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.813039    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74a7f0bd-9010-456c-a708-e0625f4e2ea2-config-out\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.913603 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.913574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-config-volume\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.913747 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.913609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.913747 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.913629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74a7f0bd-9010-456c-a708-e0625f4e2ea2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.913747 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.913647    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74a7f0bd-9010-456c-a708-e0625f4e2ea2-config-out\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.913747 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.913679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.913747 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.913707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.913747 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.913731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/74a7f0bd-9010-456c-a708-e0625f4e2ea2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.914073 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:35:11.913772    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74a7f0bd-9010-456c-a708-e0625f4e2ea2-alertmanager-trusted-ca-bundle podName:74a7f0bd-9010-456c-a708-e0625f4e2ea2 nodeName:}" failed. No retries permitted until 2026-09-16 07:35:12.413748861 +0000 UTC m=+73.340361007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/74a7f0bd-9010-456c-a708-e0625f4e2ea2-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "74a7f0bd-9010-456c-a708-e0625f4e2ea2") : configmap references non-existent config key: ca-bundle.crt
Sep 16 07:35:11.914073 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.913796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-web-config\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.914196 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.914092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74a7f0bd-9010-456c-a708-e0625f4e2ea2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.914196 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.914100    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/74a7f0bd-9010-456c-a708-e0625f4e2ea2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.914196 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.914131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.914196 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.914165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74a7f0bd-9010-456c-a708-e0625f4e2ea2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.914395 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.914197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hntv\" (UniqueName: \"kubernetes.io/projected/74a7f0bd-9010-456c-a708-e0625f4e2ea2-kube-api-access-7hntv\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.914395 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.914227    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.915147 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.915118    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74a7f0bd-9010-456c-a708-e0625f4e2ea2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.916612 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.916589    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74a7f0bd-9010-456c-a708-e0625f4e2ea2-config-out\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.916762 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.916723    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.916762 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.916748    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-config-volume\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.917245 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.917199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.917536 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.917503    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-web-config\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.917796 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.917771    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.917904 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.917812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.917972 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.917933    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74a7f0bd-9010-456c-a708-e0625f4e2ea2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.918608 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.918579    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:11.922472 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:11.922450    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hntv\" (UniqueName: \"kubernetes.io/projected/74a7f0bd-9010-456c-a708-e0625f4e2ea2-kube-api-access-7hntv\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:12.149199 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:12.149118    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tb859" event={"ID":"c906e2cd-06e6-4c74-a699-096dcc52f2b1","Type":"ContainerStarted","Data":"cff0a9809f543aea178d3be2879efe6ae3f4741cbe806f748376b955bc669b7f"}
Sep 16 07:35:12.418795 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:12.418765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74a7f0bd-9010-456c-a708-e0625f4e2ea2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:12.419647 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:12.419619    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74a7f0bd-9010-456c-a708-e0625f4e2ea2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:12.637610 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:12.637568    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:35:13.066766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:13.066744    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 07:35:13.152849 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:35:13.151769    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a7f0bd_9010_456c_a708_e0625f4e2ea2.slice/crio-6ec3c58396680818bfe621f788a7867c70eacfd724efd579499a431c75de8693 WatchSource:0}: Error finding container 6ec3c58396680818bfe621f788a7867c70eacfd724efd579499a431c75de8693: Status 404 returned error can't find the container with id 6ec3c58396680818bfe621f788a7867c70eacfd724efd579499a431c75de8693
Sep 16 07:35:14.161465 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:14.161425    2581 generic.go:356] "Generic (PLEG): container finished" podID="c906e2cd-06e6-4c74-a699-096dcc52f2b1" containerID="840de23d67a4f210e86f17be49ba5e497d3083bf961bd977f8f3055eaf903a60" exitCode=0
Sep 16 07:35:14.161931 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:14.161517    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tb859" event={"ID":"c906e2cd-06e6-4c74-a699-096dcc52f2b1","Type":"ContainerDied","Data":"840de23d67a4f210e86f17be49ba5e497d3083bf961bd977f8f3055eaf903a60"}
Sep 16 07:35:14.162988 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:14.162965    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74a7f0bd-9010-456c-a708-e0625f4e2ea2","Type":"ContainerStarted","Data":"6ec3c58396680818bfe621f788a7867c70eacfd724efd579499a431c75de8693"}
Sep 16 07:35:15.168334 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:15.168291    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tb859" event={"ID":"c906e2cd-06e6-4c74-a699-096dcc52f2b1","Type":"ContainerStarted","Data":"512044606f90a5a3bd170aed2fdf63951a36bd2ee0c54b839e09f04242dac46c"}
Sep 16 07:35:15.168769 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:15.168391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tb859" event={"ID":"c906e2cd-06e6-4c74-a699-096dcc52f2b1","Type":"ContainerStarted","Data":"85bdd6657126e0c7fa129fb0fac85e557466fa3a804faadd9f9b98706bf563cf"}
Sep 16 07:35:15.169984 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:15.169957    2581 generic.go:356] "Generic (PLEG): container finished" podID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerID="7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c" exitCode=0
Sep 16 07:35:15.170088 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:15.170028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74a7f0bd-9010-456c-a708-e0625f4e2ea2","Type":"ContainerDied","Data":"7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c"}
Sep 16 07:35:15.190263 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:15.190221    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tb859" podStartSLOduration=3.557003697 podStartE2EDuration="5.190207649s" totalImagesPullingTime="1.545948272s" totalInitContainerRuntime="87.25568ms" isStatefulPod=true podCreationTimestamp="2026-09-16 07:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:15.188626576 +0000 UTC m=+76.115238740" watchObservedRunningTime="2026-09-16 07:35:15.190207649 +0000 UTC m=+76.116819803"
Sep 16 07:35:15.420632 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:15.420564    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc"]
Sep 16 07:35:15.422404 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:15.422389    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc"
Sep 16 07:35:15.424815 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:15.424798    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 07:35:15.424963 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:15.424940    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-t8djr\""
Sep 16 07:35:15.432358 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:15.432337    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc"]
Sep 16 07:35:15.544301 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:15.544278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ncxd\" (UniqueName: \"kubernetes.io/projected/f9f23028-3891-44db-a55c-056ea019fba9-kube-api-access-6ncxd\") pod \"monitoring-plugin-5d85778b9d-2pnzc\" (UID: \"f9f23028-3891-44db-a55c-056ea019fba9\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc"
Sep 16 07:35:15.544414 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:15.544312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f9f23028-3891-44db-a55c-056ea019fba9-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-2pnzc\" (UID: \"f9f23028-3891-44db-a55c-056ea019fba9\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc"
Sep 16 07:35:15.644811 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:15.644784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ncxd\" (UniqueName: \"kubernetes.io/projected/f9f23028-3891-44db-a55c-056ea019fba9-kube-api-access-6ncxd\") pod \"monitoring-plugin-5d85778b9d-2pnzc\" (UID: \"f9f23028-3891-44db-a55c-056ea019fba9\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc"
Sep 16 07:35:15.644923 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:15.644846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f9f23028-3891-44db-a55c-056ea019fba9-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-2pnzc\" (UID: \"f9f23028-3891-44db-a55c-056ea019fba9\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc"
Sep 16 07:35:15.644966 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:35:15.644934    2581 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 07:35:15.645008 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:35:15.644999    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f23028-3891-44db-a55c-056ea019fba9-monitoring-plugin-cert podName:f9f23028-3891-44db-a55c-056ea019fba9 nodeName:}" failed. No retries permitted until 2026-09-16 07:35:16.144982993 +0000 UTC m=+77.071595124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/f9f23028-3891-44db-a55c-056ea019fba9-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-2pnzc" (UID: "f9f23028-3891-44db-a55c-056ea019fba9") : secret "monitoring-plugin-cert" not found
Sep 16 07:35:15.653808 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:15.653786    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ncxd\" (UniqueName: \"kubernetes.io/projected/f9f23028-3891-44db-a55c-056ea019fba9-kube-api-access-6ncxd\") pod \"monitoring-plugin-5d85778b9d-2pnzc\" (UID: \"f9f23028-3891-44db-a55c-056ea019fba9\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc"
Sep 16 07:35:16.149387 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:16.149345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f9f23028-3891-44db-a55c-056ea019fba9-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-2pnzc\" (UID: \"f9f23028-3891-44db-a55c-056ea019fba9\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc"
Sep 16 07:35:16.151943 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:16.151914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f9f23028-3891-44db-a55c-056ea019fba9-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-2pnzc\" (UID: \"f9f23028-3891-44db-a55c-056ea019fba9\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc"
Sep 16 07:35:16.331595 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:16.331563    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc"
Sep 16 07:35:16.870540 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:16.870487    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc"]
Sep 16 07:35:16.882794 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:35:16.882768    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f23028_3891_44db_a55c_056ea019fba9.slice/crio-dc787036c1bcef12aaf290e49a5c279a6852f7c302be0d460b5e4aba3538ded6 WatchSource:0}: Error finding container dc787036c1bcef12aaf290e49a5c279a6852f7c302be0d460b5e4aba3538ded6: Status 404 returned error can't find the container with id dc787036c1bcef12aaf290e49a5c279a6852f7c302be0d460b5e4aba3538ded6
Sep 16 07:35:17.178395 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:17.178360    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc" event={"ID":"f9f23028-3891-44db-a55c-056ea019fba9","Type":"ContainerStarted","Data":"dc787036c1bcef12aaf290e49a5c279a6852f7c302be0d460b5e4aba3538ded6"}
Sep 16 07:35:17.181578 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:17.181555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74a7f0bd-9010-456c-a708-e0625f4e2ea2","Type":"ContainerStarted","Data":"2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422"}
Sep 16 07:35:17.181675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:17.181608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74a7f0bd-9010-456c-a708-e0625f4e2ea2","Type":"ContainerStarted","Data":"652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99"}
Sep 16 07:35:17.181675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:17.181633    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74a7f0bd-9010-456c-a708-e0625f4e2ea2","Type":"ContainerStarted","Data":"b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca"}
Sep 16 07:35:17.181675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:17.181658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74a7f0bd-9010-456c-a708-e0625f4e2ea2","Type":"ContainerStarted","Data":"d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134"}
Sep 16 07:35:17.181782 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:17.181680    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74a7f0bd-9010-456c-a708-e0625f4e2ea2","Type":"ContainerStarted","Data":"a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a"}
Sep 16 07:35:18.189348 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:18.189317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74a7f0bd-9010-456c-a708-e0625f4e2ea2","Type":"ContainerStarted","Data":"3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07"}
Sep 16 07:35:18.216493 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:18.216449    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.791691632 podStartE2EDuration="7.216433918s" totalImagesPullingTime="3.406406731s" totalInitContainerRuntime="18.335555ms" isStatefulPod=false podCreationTimestamp="2026-09-16 07:35:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:18.215407609 +0000 UTC m=+79.142019765" watchObservedRunningTime="2026-09-16 07:35:18.216433918 +0000 UTC m=+79.143046070"
Sep 16 07:35:19.194758 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:19.194718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc" event={"ID":"f9f23028-3891-44db-a55c-056ea019fba9","Type":"ContainerStarted","Data":"e91a18ce88de1273a2bf07ea0f6ca0ebde9e95f7df5868fec7fa55ec06545c0b"}
Sep 16 07:35:19.195198 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:19.195025    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc"
Sep 16 07:35:19.200010 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:19.199988    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc"
Sep 16 07:35:19.208625 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:19.208581    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-2pnzc" podStartSLOduration=2.856290554 podStartE2EDuration="4.208567504s" totalImagesPullingTime="1.35227695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:19.208421049 +0000 UTC m=+80.135033204" watchObservedRunningTime="2026-09-16 07:35:19.208567504 +0000 UTC m=+80.135179656"
Sep 16 07:35:41.146336 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:35:41.146309    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h4pbq"
Sep 16 07:36:30.979676 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:30.979634    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 07:36:30.980463 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:30.980064    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="alertmanager" containerID="cri-o://a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a" gracePeriod=120
Sep 16 07:36:30.980463 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:30.980115    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="kube-rbac-proxy-metric" containerID="cri-o://2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422" gracePeriod=120
Sep 16 07:36:30.980463 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:30.980136    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="kube-rbac-proxy-web" containerID="cri-o://b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca" gracePeriod=120
Sep 16 07:36:30.980463 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:30.980181    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="prom-label-proxy" containerID="cri-o://3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07" gracePeriod=120
Sep 16 07:36:30.980463 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:30.980195    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="config-reloader" containerID="cri-o://d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134" gracePeriod=120
Sep 16 07:36:30.980463 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:30.980181    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="kube-rbac-proxy" containerID="cri-o://652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99" gracePeriod=120
Sep 16 07:36:31.439931 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:31.439905    2581 generic.go:356] "Generic (PLEG): container finished" podID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerID="3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07" exitCode=0
Sep 16 07:36:31.439931 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:31.439927    2581 generic.go:356] "Generic (PLEG): container finished" podID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerID="652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99" exitCode=0
Sep 16 07:36:31.439931 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:31.439933    2581 generic.go:356] "Generic (PLEG): container finished" podID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerID="d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134" exitCode=0
Sep 16 07:36:31.439931 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:31.439939    2581 generic.go:356] "Generic (PLEG): container finished" podID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerID="a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a" exitCode=0
Sep 16 07:36:31.440182 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:31.439982    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74a7f0bd-9010-456c-a708-e0625f4e2ea2","Type":"ContainerDied","Data":"3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07"}
Sep 16 07:36:31.440182 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:31.440058    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74a7f0bd-9010-456c-a708-e0625f4e2ea2","Type":"ContainerDied","Data":"652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99"}
Sep 16 07:36:31.440182 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:31.440087    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74a7f0bd-9010-456c-a708-e0625f4e2ea2","Type":"ContainerDied","Data":"d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134"}
Sep 16 07:36:31.440182 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:31.440113    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74a7f0bd-9010-456c-a708-e0625f4e2ea2","Type":"ContainerDied","Data":"a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a"}
Sep 16 07:36:32.216605 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.216582    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.273811 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.273778    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") "
Sep 16 07:36:32.273997 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.273852    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-config-volume\") pod \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") "
Sep 16 07:36:32.273997 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.273890    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy-web\") pod \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") "
Sep 16 07:36:32.273997 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.273927    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-web-config\") pod \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") "
Sep 16 07:36:32.273997 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.273952    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy\") pod \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") "
Sep 16 07:36:32.273997 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.273977    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/74a7f0bd-9010-456c-a708-e0625f4e2ea2-alertmanager-main-db\") pod \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") "
Sep 16 07:36:32.274248 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.274009    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74a7f0bd-9010-456c-a708-e0625f4e2ea2-alertmanager-trusted-ca-bundle\") pod \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") "
Sep 16 07:36:32.274248 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.274035    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74a7f0bd-9010-456c-a708-e0625f4e2ea2-metrics-client-ca\") pod \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") "
Sep 16 07:36:32.274248 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.274084    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-cluster-tls-config\") pod \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") "
Sep 16 07:36:32.274248 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.274112    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74a7f0bd-9010-456c-a708-e0625f4e2ea2-config-out\") pod \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") "
Sep 16 07:36:32.274248 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.274148    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74a7f0bd-9010-456c-a708-e0625f4e2ea2-tls-assets\") pod \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") "
Sep 16 07:36:32.274248 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.274178    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-main-tls\") pod \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") "
Sep 16 07:36:32.274248 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.274216    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hntv\" (UniqueName: \"kubernetes.io/projected/74a7f0bd-9010-456c-a708-e0625f4e2ea2-kube-api-access-7hntv\") pod \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\" (UID: \"74a7f0bd-9010-456c-a708-e0625f4e2ea2\") "
Sep 16 07:36:32.274574 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.274511    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a7f0bd-9010-456c-a708-e0625f4e2ea2-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "74a7f0bd-9010-456c-a708-e0625f4e2ea2" (UID: "74a7f0bd-9010-456c-a708-e0625f4e2ea2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:36:32.275116 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.274909    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a7f0bd-9010-456c-a708-e0625f4e2ea2-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "74a7f0bd-9010-456c-a708-e0625f4e2ea2" (UID: "74a7f0bd-9010-456c-a708-e0625f4e2ea2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:36:32.275453 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.275390    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a7f0bd-9010-456c-a708-e0625f4e2ea2-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "74a7f0bd-9010-456c-a708-e0625f4e2ea2" (UID: "74a7f0bd-9010-456c-a708-e0625f4e2ea2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:36:32.276943 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.276886    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-config-volume" (OuterVolumeSpecName: "config-volume") pod "74a7f0bd-9010-456c-a708-e0625f4e2ea2" (UID: "74a7f0bd-9010-456c-a708-e0625f4e2ea2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 07:36:32.277028 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.276968    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "74a7f0bd-9010-456c-a708-e0625f4e2ea2" (UID: "74a7f0bd-9010-456c-a708-e0625f4e2ea2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 07:36:32.277101 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.277053    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "74a7f0bd-9010-456c-a708-e0625f4e2ea2" (UID: "74a7f0bd-9010-456c-a708-e0625f4e2ea2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 07:36:32.278788 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.278749    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a7f0bd-9010-456c-a708-e0625f4e2ea2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "74a7f0bd-9010-456c-a708-e0625f4e2ea2" (UID: "74a7f0bd-9010-456c-a708-e0625f4e2ea2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:36:32.278897 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.278860    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a7f0bd-9010-456c-a708-e0625f4e2ea2-kube-api-access-7hntv" (OuterVolumeSpecName: "kube-api-access-7hntv") pod "74a7f0bd-9010-456c-a708-e0625f4e2ea2" (UID: "74a7f0bd-9010-456c-a708-e0625f4e2ea2"). InnerVolumeSpecName "kube-api-access-7hntv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:36:32.279092 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.279062    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a7f0bd-9010-456c-a708-e0625f4e2ea2-config-out" (OuterVolumeSpecName: "config-out") pod "74a7f0bd-9010-456c-a708-e0625f4e2ea2" (UID: "74a7f0bd-9010-456c-a708-e0625f4e2ea2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:36:32.279285 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.279261    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "74a7f0bd-9010-456c-a708-e0625f4e2ea2" (UID: "74a7f0bd-9010-456c-a708-e0625f4e2ea2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 07:36:32.279285 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.279276    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "74a7f0bd-9010-456c-a708-e0625f4e2ea2" (UID: "74a7f0bd-9010-456c-a708-e0625f4e2ea2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 07:36:32.282597 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.282574    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "74a7f0bd-9010-456c-a708-e0625f4e2ea2" (UID: "74a7f0bd-9010-456c-a708-e0625f4e2ea2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 07:36:32.287907 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.287880    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-web-config" (OuterVolumeSpecName: "web-config") pod "74a7f0bd-9010-456c-a708-e0625f4e2ea2" (UID: "74a7f0bd-9010-456c-a708-e0625f4e2ea2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 07:36:32.375391 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.375319    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74a7f0bd-9010-456c-a708-e0625f4e2ea2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:36:32.375391 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.375347    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74a7f0bd-9010-456c-a708-e0625f4e2ea2-metrics-client-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:36:32.375391 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.375357    2581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-cluster-tls-config\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:36:32.375391 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.375366    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74a7f0bd-9010-456c-a708-e0625f4e2ea2-config-out\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:36:32.375391 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.375373    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74a7f0bd-9010-456c-a708-e0625f4e2ea2-tls-assets\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:36:32.375391 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.375381    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-main-tls\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:36:32.375391 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.375389    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hntv\" (UniqueName: \"kubernetes.io/projected/74a7f0bd-9010-456c-a708-e0625f4e2ea2-kube-api-access-7hntv\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:36:32.375674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.375398    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:36:32.375674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.375407    2581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:36:32.375674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.375415    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:36:32.375674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.375422    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-web-config\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:36:32.375674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.375432    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74a7f0bd-9010-456c-a708-e0625f4e2ea2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:36:32.375674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.375439    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/74a7f0bd-9010-456c-a708-e0625f4e2ea2-alertmanager-main-db\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:36:32.448862 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.448811    2581 generic.go:356] "Generic (PLEG): container finished" podID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerID="2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422" exitCode=0
Sep 16 07:36:32.448862 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.448858    2581 generic.go:356] "Generic (PLEG): container finished" podID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerID="b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca" exitCode=0
Sep 16 07:36:32.449043 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.448896    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.449043 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.448917    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74a7f0bd-9010-456c-a708-e0625f4e2ea2","Type":"ContainerDied","Data":"2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422"}
Sep 16 07:36:32.449043 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.448988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74a7f0bd-9010-456c-a708-e0625f4e2ea2","Type":"ContainerDied","Data":"b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca"}
Sep 16 07:36:32.449043 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.449022    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74a7f0bd-9010-456c-a708-e0625f4e2ea2","Type":"ContainerDied","Data":"6ec3c58396680818bfe621f788a7867c70eacfd724efd579499a431c75de8693"}
Sep 16 07:36:32.449164 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.449052    2581 scope.go:117] "RemoveContainer" containerID="3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07"
Sep 16 07:36:32.456540 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.456521    2581 scope.go:117] "RemoveContainer" containerID="2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422"
Sep 16 07:36:32.463599 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.463581    2581 scope.go:117] "RemoveContainer" containerID="652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99"
Sep 16 07:36:32.470191 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.470168    2581 scope.go:117] "RemoveContainer" containerID="b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca"
Sep 16 07:36:32.473785 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.473762    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 07:36:32.477105 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.477084    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 07:36:32.478533 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.478517    2581 scope.go:117] "RemoveContainer" containerID="d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134"
Sep 16 07:36:32.484932 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.484915    2581 scope.go:117] "RemoveContainer" containerID="a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a"
Sep 16 07:36:32.491374 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.491355    2581 scope.go:117] "RemoveContainer" containerID="7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c"
Sep 16 07:36:32.497813 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.497793    2581 scope.go:117] "RemoveContainer" containerID="3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07"
Sep 16 07:36:32.498112 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:36:32.498093    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07\": container with ID starting with 3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07 not found: ID does not exist" containerID="3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07"
Sep 16 07:36:32.498163 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.498120    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07"} err="failed to get container status \"3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07\": rpc error: code = NotFound desc = could not find container \"3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07\": container with ID starting with 3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07 not found: ID does not exist"
Sep 16 07:36:32.498163 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.498154    2581 scope.go:117] "RemoveContainer" containerID="2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422"
Sep 16 07:36:32.498388 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:36:32.498370    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422\": container with ID starting with 2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422 not found: ID does not exist" containerID="2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422"
Sep 16 07:36:32.498448 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.498397    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422"} err="failed to get container status \"2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422\": rpc error: code = NotFound desc = could not find container \"2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422\": container with ID starting with 2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422 not found: ID does not exist"
Sep 16 07:36:32.498448 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.498420    2581 scope.go:117] "RemoveContainer" containerID="652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99"
Sep 16 07:36:32.498646 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:36:32.498629    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99\": container with ID starting with 652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99 not found: ID does not exist" containerID="652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99"
Sep 16 07:36:32.498688 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.498649    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99"} err="failed to get container status \"652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99\": rpc error: code = NotFound desc = could not find container \"652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99\": container with ID starting with 652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99 not found: ID does not exist"
Sep 16 07:36:32.498688 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.498666    2581 scope.go:117] "RemoveContainer" containerID="b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca"
Sep 16 07:36:32.499008 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:36:32.498991    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca\": container with ID starting with b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca not found: ID does not exist" containerID="b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca"
Sep 16 07:36:32.499062 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.499019    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca"} err="failed to get container status \"b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca\": rpc error: code = NotFound desc = could not find container \"b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca\": container with ID starting with b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca not found: ID does not exist"
Sep 16 07:36:32.499062 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.499032    2581 scope.go:117] "RemoveContainer" containerID="d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134"
Sep 16 07:36:32.499254 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:36:32.499241    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134\": container with ID starting with d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134 not found: ID does not exist" containerID="d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134"
Sep 16 07:36:32.499296 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.499256    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134"} err="failed to get container status \"d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134\": rpc error: code = NotFound desc = could not find container \"d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134\": container with ID starting with d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134 not found: ID does not exist"
Sep 16 07:36:32.499296 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.499267    2581 scope.go:117] "RemoveContainer" containerID="a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a"
Sep 16 07:36:32.499541 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:36:32.499519    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a\": container with ID starting with a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a not found: ID does not exist" containerID="a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a"
Sep 16 07:36:32.499618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.499549    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a"} err="failed to get container status \"a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a\": rpc error: code = NotFound desc = could not find container \"a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a\": container with ID starting with a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a not found: ID does not exist"
Sep 16 07:36:32.499618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.499568    2581 scope.go:117] "RemoveContainer" containerID="7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c"
Sep 16 07:36:32.499781 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:36:32.499764    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c\": container with ID starting with 7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c not found: ID does not exist" containerID="7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c"
Sep 16 07:36:32.499831 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.499785    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c"} err="failed to get container status \"7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c\": rpc error: code = NotFound desc = could not find container \"7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c\": container with ID starting with 7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c not found: ID does not exist"
Sep 16 07:36:32.499831 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.499798    2581 scope.go:117] "RemoveContainer" containerID="3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07"
Sep 16 07:36:32.500032 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.500015    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07"} err="failed to get container status \"3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07\": rpc error: code = NotFound desc = could not find container \"3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07\": container with ID starting with 3f8968aee0a65b094198985f2470e381d0411520b92343b26d898b68114b4c07 not found: ID does not exist"
Sep 16 07:36:32.500077 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.500033    2581 scope.go:117] "RemoveContainer" containerID="2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422"
Sep 16 07:36:32.500222 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.500207    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422"} err="failed to get container status \"2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422\": rpc error: code = NotFound desc = could not find container \"2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422\": container with ID starting with 2cb7085e8a6b2c3edf967ffcaa4d1fb4d79333961ca1894460685e04d3207422 not found: ID does not exist"
Sep 16 07:36:32.500273 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.500223    2581 scope.go:117] "RemoveContainer" containerID="652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99"
Sep 16 07:36:32.500413 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.500398    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99"} err="failed to get container status \"652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99\": rpc error: code = NotFound desc = could not find container \"652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99\": container with ID starting with 652e8291d214ed515f36af452a451b4498de386561e65b95373ab21e002a9d99 not found: ID does not exist"
Sep 16 07:36:32.500462 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.500412    2581 scope.go:117] "RemoveContainer" containerID="b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca"
Sep 16 07:36:32.500604 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.500589    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca"} err="failed to get container status \"b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca\": rpc error: code = NotFound desc = could not find container \"b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca\": container with ID starting with b5ad6e997f43957028284354662dbbf44965b10d72963752d806a6af049329ca not found: ID does not exist"
Sep 16 07:36:32.500650 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.500604    2581 scope.go:117] "RemoveContainer" containerID="d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134"
Sep 16 07:36:32.500851 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.500830    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134"} err="failed to get container status \"d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134\": rpc error: code = NotFound desc = could not find container \"d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134\": container with ID starting with d2011b0e32d724ca67943f5131d908a3ecbcca752ebc4a23ee5aa349a1989134 not found: ID does not exist"
Sep 16 07:36:32.500911 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.500853    2581 scope.go:117] "RemoveContainer" containerID="a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a"
Sep 16 07:36:32.501151 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.501121    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a"} err="failed to get container status \"a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a\": rpc error: code = NotFound desc = could not find container \"a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a\": container with ID starting with a041c0913c65c224d34170beaf2deddfdd99c780ebb2dd30438c50011894962a not found: ID does not exist"
Sep 16 07:36:32.501151 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.501146    2581 scope.go:117] "RemoveContainer" containerID="7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c"
Sep 16 07:36:32.501448 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.501417    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c"} err="failed to get container status \"7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c\": rpc error: code = NotFound desc = could not find container \"7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c\": container with ID starting with 7c26868302193dddeea802f8cc85c86071a8f8206343cecad64a1b344bc9361c not found: ID does not exist"
Sep 16 07:36:32.503331 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503312    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 07:36:32.503557 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503546    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="init-config-reloader"
Sep 16 07:36:32.503592 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503558    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="init-config-reloader"
Sep 16 07:36:32.503592 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503568    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="alertmanager"
Sep 16 07:36:32.503592 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503574    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="alertmanager"
Sep 16 07:36:32.503592 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503579    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="kube-rbac-proxy"
Sep 16 07:36:32.503592 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503585    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="kube-rbac-proxy"
Sep 16 07:36:32.503741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503595    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="config-reloader"
Sep 16 07:36:32.503741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503600    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="config-reloader"
Sep 16 07:36:32.503741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503612    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="kube-rbac-proxy-web"
Sep 16 07:36:32.503741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503617    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="kube-rbac-proxy-web"
Sep 16 07:36:32.503741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503623    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="kube-rbac-proxy-metric"
Sep 16 07:36:32.503741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503628    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="kube-rbac-proxy-metric"
Sep 16 07:36:32.503741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503634    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="prom-label-proxy"
Sep 16 07:36:32.503741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503638    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="prom-label-proxy"
Sep 16 07:36:32.503741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503673    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="prom-label-proxy"
Sep 16 07:36:32.503741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503680    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="alertmanager"
Sep 16 07:36:32.503741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503685    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="kube-rbac-proxy-web"
Sep 16 07:36:32.503741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503691    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="config-reloader"
Sep 16 07:36:32.503741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503695    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="kube-rbac-proxy"
Sep 16 07:36:32.503741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.503700    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" containerName="kube-rbac-proxy-metric"
Sep 16 07:36:32.508471 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.508456    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.511086 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.510943    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 07:36:32.511086 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.510969    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 07:36:32.511086 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.510971    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 07:36:32.511086 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.511016    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 07:36:32.511086 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.511040    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 07:36:32.511086 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.510977    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 07:36:32.511086 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.511041    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 07:36:32.511086 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.511087    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 07:36:32.511386 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.511016    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-s8rxs\""
Sep 16 07:36:32.519414 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.519393    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 07:36:32.526289 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.526270    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 07:36:32.576871 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.576837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.576991 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.576879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9760136-8a27-4247-bf74-e95bd537b8f2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.576991 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.576905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.576991 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.576942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9760136-8a27-4247-bf74-e95bd537b8f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.576991 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.576963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-config-volume\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.576991 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.576985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.577158 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.576999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-web-config\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.577158 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.577042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9760136-8a27-4247-bf74-e95bd537b8f2-config-out\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.577158 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.577091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e9760136-8a27-4247-bf74-e95bd537b8f2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.577158 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.577144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.577277 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.577163    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.577277 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.577180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksfnv\" (UniqueName: \"kubernetes.io/projected/e9760136-8a27-4247-bf74-e95bd537b8f2-kube-api-access-ksfnv\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.577277 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.577216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9760136-8a27-4247-bf74-e95bd537b8f2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.678417 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.678386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.678417 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.678419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.678591 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.678436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksfnv\" (UniqueName: \"kubernetes.io/projected/e9760136-8a27-4247-bf74-e95bd537b8f2-kube-api-access-ksfnv\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.678591 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.678454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9760136-8a27-4247-bf74-e95bd537b8f2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.678591 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.678562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.678713 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.678601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9760136-8a27-4247-bf74-e95bd537b8f2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.678713 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.678649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.678713 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.678679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9760136-8a27-4247-bf74-e95bd537b8f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.678914 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.678718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-config-volume\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.678914 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.678758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.678914 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.678782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-web-config\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.678914 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.678810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9760136-8a27-4247-bf74-e95bd537b8f2-config-out\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.678914 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.678854    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e9760136-8a27-4247-bf74-e95bd537b8f2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.679375 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.679353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e9760136-8a27-4247-bf74-e95bd537b8f2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.679462 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.679441    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9760136-8a27-4247-bf74-e95bd537b8f2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.679810 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.679778    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9760136-8a27-4247-bf74-e95bd537b8f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.681785 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.681564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.681785 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.681611    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9760136-8a27-4247-bf74-e95bd537b8f2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.681785 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.681642    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.681785 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.681657    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.681785 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.681745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.682075 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.682054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-config-volume\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.682111 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.682054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-web-config\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.682383 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.682362    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9760136-8a27-4247-bf74-e95bd537b8f2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.683347 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.683330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9760136-8a27-4247-bf74-e95bd537b8f2-config-out\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.686357 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.686337    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksfnv\" (UniqueName: \"kubernetes.io/projected/e9760136-8a27-4247-bf74-e95bd537b8f2-kube-api-access-ksfnv\") pod \"alertmanager-main-0\" (UID: \"e9760136-8a27-4247-bf74-e95bd537b8f2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:32.819592 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:32.819559    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 07:36:33.095781 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:33.095751    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 07:36:33.096550 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:36:33.096527    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9760136_8a27_4247_bf74_e95bd537b8f2.slice/crio-732c25fe71a31026712ff9b751a4053ee55d864a9507f697407230852dfc53fd WatchSource:0}: Error finding container 732c25fe71a31026712ff9b751a4053ee55d864a9507f697407230852dfc53fd: Status 404 returned error can't find the container with id 732c25fe71a31026712ff9b751a4053ee55d864a9507f697407230852dfc53fd
Sep 16 07:36:33.454160 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:33.454126    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9760136-8a27-4247-bf74-e95bd537b8f2" containerID="d0fc6243d1fadf391016601124bf03c8e34f9a11376ed2b1a3c5ad5ee382a6df" exitCode=0
Sep 16 07:36:33.454577 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:33.454210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9760136-8a27-4247-bf74-e95bd537b8f2","Type":"ContainerDied","Data":"d0fc6243d1fadf391016601124bf03c8e34f9a11376ed2b1a3c5ad5ee382a6df"}
Sep 16 07:36:33.454577 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:33.454292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9760136-8a27-4247-bf74-e95bd537b8f2","Type":"ContainerStarted","Data":"732c25fe71a31026712ff9b751a4053ee55d864a9507f697407230852dfc53fd"}
Sep 16 07:36:33.682904 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:33.682879    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74a7f0bd-9010-456c-a708-e0625f4e2ea2" path="/var/lib/kubelet/pods/74a7f0bd-9010-456c-a708-e0625f4e2ea2/volumes"
Sep 16 07:36:34.463648 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:34.463603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9760136-8a27-4247-bf74-e95bd537b8f2","Type":"ContainerStarted","Data":"feb35f78e3ec05dbc43777460a5f2c51c0cdbaca76bee101266516fb33104e18"}
Sep 16 07:36:34.464049 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:34.463698    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9760136-8a27-4247-bf74-e95bd537b8f2","Type":"ContainerStarted","Data":"1508a7a3caff615e79e6f923ed9fd64afc61a2af114d20a8f137900da94f3cef"}
Sep 16 07:36:34.464049 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:34.463755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9760136-8a27-4247-bf74-e95bd537b8f2","Type":"ContainerStarted","Data":"3ac03f53a16321edaba5394d7bd26d8400fa0cb87baafea9482cf2c5ed8815cf"}
Sep 16 07:36:34.464049 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:34.463803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9760136-8a27-4247-bf74-e95bd537b8f2","Type":"ContainerStarted","Data":"2d33f35e121d0cfe1cf4802065ba052de2d7c6a6f602d94e5ec7d1c708fb2bad"}
Sep 16 07:36:34.464049 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:34.463885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9760136-8a27-4247-bf74-e95bd537b8f2","Type":"ContainerStarted","Data":"817f1bb5f7ad838dd7d70b6efe8d67c013085f7937f4ff60925c81e341d29dbd"}
Sep 16 07:36:34.464049 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:34.463933    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9760136-8a27-4247-bf74-e95bd537b8f2","Type":"ContainerStarted","Data":"698b3b12a634950a3f39bad229b1161ae921fd70983606b3145a468d51abf4fb"}
Sep 16 07:36:34.495283 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:34.495235    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.476761465 podStartE2EDuration="2.495219676s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.458211ms" isStatefulPod=false podCreationTimestamp="2026-09-16 07:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:36:34.493234828 +0000 UTC m=+155.419847005" watchObservedRunningTime="2026-09-16 07:36:34.495219676 +0000 UTC m=+155.421831828"
Sep 16 07:36:35.000597 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.000566    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"]
Sep 16 07:36:35.003780 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.003765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.006517 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.006494    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xtpdw\""
Sep 16 07:36:35.006517 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.006517    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 07:36:35.006710 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.006529    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 07:36:35.006710 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.006533    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 07:36:35.006710 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.006581    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 07:36:35.006710 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.006670    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 07:36:35.011649 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.011627    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 07:36:35.014572 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.014552    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"]
Sep 16 07:36:35.096375 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.096350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd361603-bf40-4600-88d1-652c1f84158a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.096375 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.096379    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd361603-bf40-4600-88d1-652c1f84158a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.096557 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.096401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bd361603-bf40-4600-88d1-652c1f84158a-secret-telemeter-client\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.096557 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.096469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bd361603-bf40-4600-88d1-652c1f84158a-federate-client-tls\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.096557 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.096508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmllq\" (UniqueName: \"kubernetes.io/projected/bd361603-bf40-4600-88d1-652c1f84158a-kube-api-access-nmllq\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.096557 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.096526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bd361603-bf40-4600-88d1-652c1f84158a-telemeter-client-tls\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.096690 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.096557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd361603-bf40-4600-88d1-652c1f84158a-serving-certs-ca-bundle\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.096690 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.096582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd361603-bf40-4600-88d1-652c1f84158a-metrics-client-ca\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.197135 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.197104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bd361603-bf40-4600-88d1-652c1f84158a-federate-client-tls\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.197250 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.197148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmllq\" (UniqueName: \"kubernetes.io/projected/bd361603-bf40-4600-88d1-652c1f84158a-kube-api-access-nmllq\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.197250 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.197170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bd361603-bf40-4600-88d1-652c1f84158a-telemeter-client-tls\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.197250 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.197207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd361603-bf40-4600-88d1-652c1f84158a-serving-certs-ca-bundle\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.197250 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.197231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd361603-bf40-4600-88d1-652c1f84158a-metrics-client-ca\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.197424 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.197257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd361603-bf40-4600-88d1-652c1f84158a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.197424 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.197284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd361603-bf40-4600-88d1-652c1f84158a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.197525 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.197442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bd361603-bf40-4600-88d1-652c1f84158a-secret-telemeter-client\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.198093 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.198064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd361603-bf40-4600-88d1-652c1f84158a-metrics-client-ca\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.198190 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.198164    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd361603-bf40-4600-88d1-652c1f84158a-serving-certs-ca-bundle\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.198264 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.198247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd361603-bf40-4600-88d1-652c1f84158a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.199761 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.199735    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bd361603-bf40-4600-88d1-652c1f84158a-federate-client-tls\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.199851 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.199801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bd361603-bf40-4600-88d1-652c1f84158a-secret-telemeter-client\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.200019 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.200002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd361603-bf40-4600-88d1-652c1f84158a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.200019 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.200013    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bd361603-bf40-4600-88d1-652c1f84158a-telemeter-client-tls\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.204582 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.204564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmllq\" (UniqueName: \"kubernetes.io/projected/bd361603-bf40-4600-88d1-652c1f84158a-kube-api-access-nmllq\") pod \"telemeter-client-5fdc6b46f8-2vpvx\" (UID: \"bd361603-bf40-4600-88d1-652c1f84158a\") " pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.316086 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.315982    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"
Sep 16 07:36:35.600441 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:35.600364    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx"]
Sep 16 07:36:35.603559 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:36:35.603531    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd361603_bf40_4600_88d1_652c1f84158a.slice/crio-a4f072636684137d5cdf58933f526fe9b506c2d83a34896980c989887bfa789f WatchSource:0}: Error finding container a4f072636684137d5cdf58933f526fe9b506c2d83a34896980c989887bfa789f: Status 404 returned error can't find the container with id a4f072636684137d5cdf58933f526fe9b506c2d83a34896980c989887bfa789f
Sep 16 07:36:36.473330 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:36.473299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx" event={"ID":"bd361603-bf40-4600-88d1-652c1f84158a","Type":"ContainerStarted","Data":"a4f072636684137d5cdf58933f526fe9b506c2d83a34896980c989887bfa789f"}
Sep 16 07:36:37.482886 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:37.482856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx" event={"ID":"bd361603-bf40-4600-88d1-652c1f84158a","Type":"ContainerStarted","Data":"fc3ba270f032583d5346c69ff861f132ffbef75f20cc525ee730b884d95bf629"}
Sep 16 07:36:37.483258 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:37.482936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx" event={"ID":"bd361603-bf40-4600-88d1-652c1f84158a","Type":"ContainerStarted","Data":"cf42c9bcbfc45a931b8a78426cf6330b856d23727a00429b2204135c95a43635"}
Sep 16 07:36:38.488910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:38.488872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx" event={"ID":"bd361603-bf40-4600-88d1-652c1f84158a","Type":"ContainerStarted","Data":"5a9f6ee4ff98866558834bff723d662716019de26b341a7b5253dcd9b8181181"}
Sep 16 07:36:38.510317 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:36:38.510267    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5fdc6b46f8-2vpvx" podStartSLOduration=2.786140757 podStartE2EDuration="4.510253668s" totalImagesPullingTime="1.724112911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:36:38.509726254 +0000 UTC m=+159.436338418" watchObservedRunningTime="2026-09-16 07:36:38.510253668 +0000 UTC m=+159.436865831"
Sep 16 07:38:59.471677 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:38:59.471642    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 07:38:59.472368 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:38:59.472352    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 07:38:59.477850 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:38:59.477830    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 07:39:38.650653 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:38.650620    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2ttrm"]
Sep 16 07:39:38.654009 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:38.653993    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ttrm"
Sep 16 07:39:38.656545 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:38.656513    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 07:39:38.656668 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:38.656603    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 07:39:38.657383 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:38.657368    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6w7ts\""
Sep 16 07:39:38.660891 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:38.660867    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2ttrm"]
Sep 16 07:39:38.761788 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:38.761758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgtrq\" (UniqueName: \"kubernetes.io/projected/69a2e977-6cc3-42cb-b3df-18bb5c00d3a6-kube-api-access-fgtrq\") pod \"cert-manager-cainjector-559cdd8c87-2ttrm\" (UID: \"69a2e977-6cc3-42cb-b3df-18bb5c00d3a6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ttrm"
Sep 16 07:39:38.761921 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:38.761798    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69a2e977-6cc3-42cb-b3df-18bb5c00d3a6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2ttrm\" (UID: \"69a2e977-6cc3-42cb-b3df-18bb5c00d3a6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ttrm"
Sep 16 07:39:38.862797 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:38.862772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtrq\" (UniqueName: \"kubernetes.io/projected/69a2e977-6cc3-42cb-b3df-18bb5c00d3a6-kube-api-access-fgtrq\") pod \"cert-manager-cainjector-559cdd8c87-2ttrm\" (UID: \"69a2e977-6cc3-42cb-b3df-18bb5c00d3a6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ttrm"
Sep 16 07:39:38.862912 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:38.862805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69a2e977-6cc3-42cb-b3df-18bb5c00d3a6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2ttrm\" (UID: \"69a2e977-6cc3-42cb-b3df-18bb5c00d3a6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ttrm"
Sep 16 07:39:38.870326 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:38.870307    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69a2e977-6cc3-42cb-b3df-18bb5c00d3a6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2ttrm\" (UID: \"69a2e977-6cc3-42cb-b3df-18bb5c00d3a6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ttrm"
Sep 16 07:39:38.870486 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:38.870446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtrq\" (UniqueName: \"kubernetes.io/projected/69a2e977-6cc3-42cb-b3df-18bb5c00d3a6-kube-api-access-fgtrq\") pod \"cert-manager-cainjector-559cdd8c87-2ttrm\" (UID: \"69a2e977-6cc3-42cb-b3df-18bb5c00d3a6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ttrm"
Sep 16 07:39:38.962546 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:38.962525    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ttrm"
Sep 16 07:39:39.227902 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:39.227872    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2ttrm"]
Sep 16 07:39:39.230264 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:39:39.230239    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69a2e977_6cc3_42cb_b3df_18bb5c00d3a6.slice/crio-ab009e3e0867e2100b75aca5b2435749cb936467ac9a448f5a469f568645626a WatchSource:0}: Error finding container ab009e3e0867e2100b75aca5b2435749cb936467ac9a448f5a469f568645626a: Status 404 returned error can't find the container with id ab009e3e0867e2100b75aca5b2435749cb936467ac9a448f5a469f568645626a
Sep 16 07:39:39.232250 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:39.232228    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 07:39:40.115854 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:40.115801    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ttrm" event={"ID":"69a2e977-6cc3-42cb-b3df-18bb5c00d3a6","Type":"ContainerStarted","Data":"ab009e3e0867e2100b75aca5b2435749cb936467ac9a448f5a469f568645626a"}
Sep 16 07:39:44.134618 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:44.134579    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ttrm" event={"ID":"69a2e977-6cc3-42cb-b3df-18bb5c00d3a6","Type":"ContainerStarted","Data":"b7d3118bc223ad93a02270221160e8d33c46fed69b345acfa32ef8ed9ce416ca"}
Sep 16 07:39:44.148297 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:39:44.148248    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ttrm" podStartSLOduration=1.9072109149999998 podStartE2EDuration="6.148234312s" totalImagesPullingTime="4.241023397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:39:44.147523795 +0000 UTC m=+345.074135947" watchObservedRunningTime="2026-09-16 07:39:44.148234312 +0000 UTC m=+345.074846465"
Sep 16 07:40:12.578252 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.578172    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"]
Sep 16 07:40:12.584501 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.584483    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.587086 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.587062    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 07:40:12.587212 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.587179    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 07:40:12.587212 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.587200    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cljxh\""
Sep 16 07:40:12.587438 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.587423    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 07:40:12.587706 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.587693    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 07:40:12.594741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.594717    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"]
Sep 16 07:40:12.681875 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.681847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/248a1a40-3a78-42ff-969b-cdcd9785711e-webhook-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-wv6kt\" (UID: \"248a1a40-3a78-42ff-969b-cdcd9785711e\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.682010 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.681888    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/248a1a40-3a78-42ff-969b-cdcd9785711e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-wv6kt\" (UID: \"248a1a40-3a78-42ff-969b-cdcd9785711e\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.682010 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.681946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzdgh\" (UniqueName: \"kubernetes.io/projected/248a1a40-3a78-42ff-969b-cdcd9785711e-kube-api-access-kzdgh\") pod \"opendatahub-operator-controller-manager-dcb658945-wv6kt\" (UID: \"248a1a40-3a78-42ff-969b-cdcd9785711e\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.682010 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.681990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/248a1a40-3a78-42ff-969b-cdcd9785711e-manifests\") pod \"opendatahub-operator-controller-manager-dcb658945-wv6kt\" (UID: \"248a1a40-3a78-42ff-969b-cdcd9785711e\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.682163 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.682079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/248a1a40-3a78-42ff-969b-cdcd9785711e-tmp\") pod \"opendatahub-operator-controller-manager-dcb658945-wv6kt\" (UID: \"248a1a40-3a78-42ff-969b-cdcd9785711e\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.783196 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.783160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/248a1a40-3a78-42ff-969b-cdcd9785711e-manifests\") pod \"opendatahub-operator-controller-manager-dcb658945-wv6kt\" (UID: \"248a1a40-3a78-42ff-969b-cdcd9785711e\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.783348 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.783218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/248a1a40-3a78-42ff-969b-cdcd9785711e-tmp\") pod \"opendatahub-operator-controller-manager-dcb658945-wv6kt\" (UID: \"248a1a40-3a78-42ff-969b-cdcd9785711e\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.783348 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.783239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/248a1a40-3a78-42ff-969b-cdcd9785711e-webhook-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-wv6kt\" (UID: \"248a1a40-3a78-42ff-969b-cdcd9785711e\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.783348 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.783259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/248a1a40-3a78-42ff-969b-cdcd9785711e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-wv6kt\" (UID: \"248a1a40-3a78-42ff-969b-cdcd9785711e\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.783348 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.783286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzdgh\" (UniqueName: \"kubernetes.io/projected/248a1a40-3a78-42ff-969b-cdcd9785711e-kube-api-access-kzdgh\") pod \"opendatahub-operator-controller-manager-dcb658945-wv6kt\" (UID: \"248a1a40-3a78-42ff-969b-cdcd9785711e\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.783624 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.783600    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/248a1a40-3a78-42ff-969b-cdcd9785711e-manifests\") pod \"opendatahub-operator-controller-manager-dcb658945-wv6kt\" (UID: \"248a1a40-3a78-42ff-969b-cdcd9785711e\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.785589 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.785557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/248a1a40-3a78-42ff-969b-cdcd9785711e-tmp\") pod \"opendatahub-operator-controller-manager-dcb658945-wv6kt\" (UID: \"248a1a40-3a78-42ff-969b-cdcd9785711e\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.785701 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.785665    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/248a1a40-3a78-42ff-969b-cdcd9785711e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-wv6kt\" (UID: \"248a1a40-3a78-42ff-969b-cdcd9785711e\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.785866 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.785846    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/248a1a40-3a78-42ff-969b-cdcd9785711e-webhook-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-wv6kt\" (UID: \"248a1a40-3a78-42ff-969b-cdcd9785711e\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.793560 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.793536    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzdgh\" (UniqueName: \"kubernetes.io/projected/248a1a40-3a78-42ff-969b-cdcd9785711e-kube-api-access-kzdgh\") pod \"opendatahub-operator-controller-manager-dcb658945-wv6kt\" (UID: \"248a1a40-3a78-42ff-969b-cdcd9785711e\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:12.895003 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:12.894933    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:13.166465 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:13.166442    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"]
Sep 16 07:40:13.168798 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:40:13.168767    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248a1a40_3a78_42ff_969b_cdcd9785711e.slice/crio-423d6749f970d3d682f4ec3b5b8fc37530313e5a7ca5b4a7c156ffa3c7fbcfdf WatchSource:0}: Error finding container 423d6749f970d3d682f4ec3b5b8fc37530313e5a7ca5b4a7c156ffa3c7fbcfdf: Status 404 returned error can't find the container with id 423d6749f970d3d682f4ec3b5b8fc37530313e5a7ca5b4a7c156ffa3c7fbcfdf
Sep 16 07:40:13.243403 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:13.243369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt" event={"ID":"248a1a40-3a78-42ff-969b-cdcd9785711e","Type":"ContainerStarted","Data":"423d6749f970d3d682f4ec3b5b8fc37530313e5a7ca5b4a7c156ffa3c7fbcfdf"}
Sep 16 07:40:16.258310 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:16.258282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt" event={"ID":"248a1a40-3a78-42ff-969b-cdcd9785711e","Type":"ContainerStarted","Data":"630b01904e39dfa477b0dceb5d5a3cc11eb7861776c7a523c0ab13159613e518"}
Sep 16 07:40:17.264083 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:17.264057    2581 generic.go:356] "Generic (PLEG): container finished" podID="248a1a40-3a78-42ff-969b-cdcd9785711e" containerID="630b01904e39dfa477b0dceb5d5a3cc11eb7861776c7a523c0ab13159613e518" exitCode=0
Sep 16 07:40:17.264397 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:17.264131    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt" event={"ID":"248a1a40-3a78-42ff-969b-cdcd9785711e","Type":"ContainerDied","Data":"630b01904e39dfa477b0dceb5d5a3cc11eb7861776c7a523c0ab13159613e518"}
Sep 16 07:40:18.270054 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:18.270015    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt" event={"ID":"248a1a40-3a78-42ff-969b-cdcd9785711e","Type":"ContainerStarted","Data":"0b84186232aba6bbb2a78cd0947c26f46e5a571d4f02137c9922c3f478516b2c"}
Sep 16 07:40:18.270424 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:18.270126    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:18.291307 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:18.291256    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt" podStartSLOduration=1.804905836 podStartE2EDuration="6.291243189s" totalImagesPullingTime="3.454258577s" totalInitContainerRuntime="1.032078776s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:40:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:40:18.290869053 +0000 UTC m=+379.217481206" watchObservedRunningTime="2026-09-16 07:40:18.291243189 +0000 UTC m=+379.217855341"
Sep 16 07:40:29.275652 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:29.275623    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-wv6kt"
Sep 16 07:40:33.362647 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.362606    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5"]
Sep 16 07:40:33.371043 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.371018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5"
Sep 16 07:40:33.374701 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.374677    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 07:40:33.375041 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.375022    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 07:40:33.375420 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.375400    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 07:40:33.377943 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.377923    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 07:40:33.382480 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.382460    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5"]
Sep 16 07:40:33.432976 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.432953    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de638055-4af0-4f62-9181-de3e456e54fe-tls-certs\") pod \"kube-auth-proxy-7b6965477b-hhrj5\" (UID: \"de638055-4af0-4f62-9181-de3e456e54fe\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5"
Sep 16 07:40:33.433109 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.432984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55mhk\" (UniqueName: \"kubernetes.io/projected/de638055-4af0-4f62-9181-de3e456e54fe-kube-api-access-55mhk\") pod \"kube-auth-proxy-7b6965477b-hhrj5\" (UID: \"de638055-4af0-4f62-9181-de3e456e54fe\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5"
Sep 16 07:40:33.433109 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.433032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de638055-4af0-4f62-9181-de3e456e54fe-tmp\") pod \"kube-auth-proxy-7b6965477b-hhrj5\" (UID: \"de638055-4af0-4f62-9181-de3e456e54fe\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5"
Sep 16 07:40:33.533476 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.533440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de638055-4af0-4f62-9181-de3e456e54fe-tls-certs\") pod \"kube-auth-proxy-7b6965477b-hhrj5\" (UID: \"de638055-4af0-4f62-9181-de3e456e54fe\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5"
Sep 16 07:40:33.533617 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.533482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55mhk\" (UniqueName: \"kubernetes.io/projected/de638055-4af0-4f62-9181-de3e456e54fe-kube-api-access-55mhk\") pod \"kube-auth-proxy-7b6965477b-hhrj5\" (UID: \"de638055-4af0-4f62-9181-de3e456e54fe\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5"
Sep 16 07:40:33.533617 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.533554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de638055-4af0-4f62-9181-de3e456e54fe-tmp\") pod \"kube-auth-proxy-7b6965477b-hhrj5\" (UID: \"de638055-4af0-4f62-9181-de3e456e54fe\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5"
Sep 16 07:40:33.535849 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.535806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de638055-4af0-4f62-9181-de3e456e54fe-tmp\") pod \"kube-auth-proxy-7b6965477b-hhrj5\" (UID: \"de638055-4af0-4f62-9181-de3e456e54fe\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5"
Sep 16 07:40:33.535962 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.535900    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de638055-4af0-4f62-9181-de3e456e54fe-tls-certs\") pod \"kube-auth-proxy-7b6965477b-hhrj5\" (UID: \"de638055-4af0-4f62-9181-de3e456e54fe\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5"
Sep 16 07:40:33.540917 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.540892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55mhk\" (UniqueName: \"kubernetes.io/projected/de638055-4af0-4f62-9181-de3e456e54fe-kube-api-access-55mhk\") pod \"kube-auth-proxy-7b6965477b-hhrj5\" (UID: \"de638055-4af0-4f62-9181-de3e456e54fe\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5"
Sep 16 07:40:33.681748 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.681722    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5"
Sep 16 07:40:33.969666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:33.969585    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5"]
Sep 16 07:40:33.973288 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:40:33.973265    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde638055_4af0_4f62_9181_de3e456e54fe.slice/crio-4659ca3d0ada308beb285a2cd40bdf790fce2223b89a32bab8500ff840388259 WatchSource:0}: Error finding container 4659ca3d0ada308beb285a2cd40bdf790fce2223b89a32bab8500ff840388259: Status 404 returned error can't find the container with id 4659ca3d0ada308beb285a2cd40bdf790fce2223b89a32bab8500ff840388259
Sep 16 07:40:34.330767 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:34.330613    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5" event={"ID":"de638055-4af0-4f62-9181-de3e456e54fe","Type":"ContainerStarted","Data":"4659ca3d0ada308beb285a2cd40bdf790fce2223b89a32bab8500ff840388259"}
Sep 16 07:40:37.344701 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:37.344664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5" event={"ID":"de638055-4af0-4f62-9181-de3e456e54fe","Type":"ContainerStarted","Data":"57769f6e0d6c143d26145018fc3c51840d6075a8a40c3f291f16f344c7f1cc03"}
Sep 16 07:40:37.360160 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:37.360116    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7b6965477b-hhrj5" podStartSLOduration=1.71709472 podStartE2EDuration="4.360100667s" totalImagesPullingTime="2.643005947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:40:37.358717623 +0000 UTC m=+398.285329775" watchObservedRunningTime="2026-09-16 07:40:37.360100667 +0000 UTC m=+398.286712820"
Sep 16 07:40:39.600078 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.600040    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"]
Sep 16 07:40:39.603576 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.603558    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.605966 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.605923    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 07:40:39.606172 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.605966    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mpwmg\""
Sep 16 07:40:39.616789 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.616768    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"]
Sep 16 07:40:39.680165 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.680139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.680294 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.680173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.680294 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.680212    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.680294 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.680242    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.680294 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.680261    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.680294 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.680285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.680473 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.680354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkkbj\" (UniqueName: \"kubernetes.io/projected/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-kube-api-access-kkkbj\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.680473 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.680414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.680473 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.680463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.781718 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.781693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkkbj\" (UniqueName: \"kubernetes.io/projected/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-kube-api-access-kkkbj\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.781917 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.781735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.781917 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.781765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.781917 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.781808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.781917 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.781859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.781917 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.781889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.782210 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.781938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.782210 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.781975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.782210 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.782009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.782368 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.782238    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.782368 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.782252    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.782522 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.782500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.782621 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.782598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.782674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.782652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.784272 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.784250    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.784510 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.784490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.795788 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.795765    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.799886 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.799862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkkbj\" (UniqueName: \"kubernetes.io/projected/b4e2eab0-2feb-42c1-b4d4-386e95a6e661-kube-api-access-kkkbj\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh\" (UID: \"b4e2eab0-2feb-42c1-b4d4-386e95a6e661\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:39.915097 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:39.915071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:40.187048 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:40.187026    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"]
Sep 16 07:40:40.189694 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:40:40.189666    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e2eab0_2feb_42c1_b4d4_386e95a6e661.slice/crio-71b074ce418f7be54ab70d497581a5b2a2988b3b81e50a5ae0c0ab7b103dbb50 WatchSource:0}: Error finding container 71b074ce418f7be54ab70d497581a5b2a2988b3b81e50a5ae0c0ab7b103dbb50: Status 404 returned error can't find the container with id 71b074ce418f7be54ab70d497581a5b2a2988b3b81e50a5ae0c0ab7b103dbb50
Sep 16 07:40:40.356365 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:40.356331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh" event={"ID":"b4e2eab0-2feb-42c1-b4d4-386e95a6e661","Type":"ContainerStarted","Data":"71b074ce418f7be54ab70d497581a5b2a2988b3b81e50a5ae0c0ab7b103dbb50"}
Sep 16 07:40:43.811133 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:43.811101    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cnff7"]
Sep 16 07:40:43.815137 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:43.815117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-cnff7"
Sep 16 07:40:43.818943 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:43.818915    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 16 07:40:43.819052 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:43.818929    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wsjqx\""
Sep 16 07:40:43.838135 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:43.838101    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cnff7"]
Sep 16 07:40:43.915930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:43.915896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgmqd\" (UniqueName: \"kubernetes.io/projected/251f9852-5b84-433f-b98f-6728265a094d-kube-api-access-qgmqd\") pod \"odh-model-controller-7998c67dc4-cnff7\" (UID: \"251f9852-5b84-433f-b98f-6728265a094d\") " pod="opendatahub/odh-model-controller-7998c67dc4-cnff7"
Sep 16 07:40:43.916087 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:43.915949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/251f9852-5b84-433f-b98f-6728265a094d-cert\") pod \"odh-model-controller-7998c67dc4-cnff7\" (UID: \"251f9852-5b84-433f-b98f-6728265a094d\") " pod="opendatahub/odh-model-controller-7998c67dc4-cnff7"
Sep 16 07:40:44.017537 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:44.017495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgmqd\" (UniqueName: \"kubernetes.io/projected/251f9852-5b84-433f-b98f-6728265a094d-kube-api-access-qgmqd\") pod \"odh-model-controller-7998c67dc4-cnff7\" (UID: \"251f9852-5b84-433f-b98f-6728265a094d\") " pod="opendatahub/odh-model-controller-7998c67dc4-cnff7"
Sep 16 07:40:44.017725 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:44.017541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/251f9852-5b84-433f-b98f-6728265a094d-cert\") pod \"odh-model-controller-7998c67dc4-cnff7\" (UID: \"251f9852-5b84-433f-b98f-6728265a094d\") " pod="opendatahub/odh-model-controller-7998c67dc4-cnff7"
Sep 16 07:40:44.019950 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:44.019918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/251f9852-5b84-433f-b98f-6728265a094d-cert\") pod \"odh-model-controller-7998c67dc4-cnff7\" (UID: \"251f9852-5b84-433f-b98f-6728265a094d\") " pod="opendatahub/odh-model-controller-7998c67dc4-cnff7"
Sep 16 07:40:44.024680 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:44.024654    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgmqd\" (UniqueName: \"kubernetes.io/projected/251f9852-5b84-433f-b98f-6728265a094d-kube-api-access-qgmqd\") pod \"odh-model-controller-7998c67dc4-cnff7\" (UID: \"251f9852-5b84-433f-b98f-6728265a094d\") " pod="opendatahub/odh-model-controller-7998c67dc4-cnff7"
Sep 16 07:40:44.127576 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:44.127486    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-cnff7"
Sep 16 07:40:44.395551 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:44.395441    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cnff7"]
Sep 16 07:40:44.398703 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:40:44.398672    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251f9852_5b84_433f_b98f_6728265a094d.slice/crio-3de1ec4408974a2a9b3d8d0896b302a5041d5a9f7e464b79e60d6bf2daa0b5fa WatchSource:0}: Error finding container 3de1ec4408974a2a9b3d8d0896b302a5041d5a9f7e464b79e60d6bf2daa0b5fa: Status 404 returned error can't find the container with id 3de1ec4408974a2a9b3d8d0896b302a5041d5a9f7e464b79e60d6bf2daa0b5fa
Sep 16 07:40:45.378515 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:45.378457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-cnff7" event={"ID":"251f9852-5b84-433f-b98f-6728265a094d","Type":"ContainerStarted","Data":"3de1ec4408974a2a9b3d8d0896b302a5041d5a9f7e464b79e60d6bf2daa0b5fa"}
Sep 16 07:40:48.392557 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:48.392522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-cnff7" event={"ID":"251f9852-5b84-433f-b98f-6728265a094d","Type":"ContainerStarted","Data":"df2fac7491369841c76df46338a1b18c1de1e5f165a5154f172b5625217dc56c"}
Sep 16 07:40:48.393040 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:48.392662    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-cnff7"
Sep 16 07:40:48.408327 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:48.408278    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-cnff7" podStartSLOduration=2.373848116 podStartE2EDuration="5.408258867s" totalImagesPullingTime="3.034410751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:40:48.406868201 +0000 UTC m=+409.333480354" watchObservedRunningTime="2026-09-16 07:40:48.408258867 +0000 UTC m=+409.334871021"
Sep 16 07:40:49.741726 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:49.741678    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:40:49.742036 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:49.741774    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:40:49.742036 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:49.741839    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:40:50.405554 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:50.405518    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh" event={"ID":"b4e2eab0-2feb-42c1-b4d4-386e95a6e661","Type":"ContainerStarted","Data":"b8cfa789514dedb14ee7c33cc73c4ae5c9a51cbf9b7e74447d14829aa9516f6c"}
Sep 16 07:40:50.440794 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:50.440743    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh" podStartSLOduration=1.8910499330000001 podStartE2EDuration="11.440727631s" totalImagesPullingTime="9.549677698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:40:50.440230443 +0000 UTC m=+411.366842594" watchObservedRunningTime="2026-09-16 07:40:50.440727631 +0000 UTC m=+411.367339783"
Sep 16 07:40:50.915884 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:50.915856    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:50.920598 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:50.920574    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:51.211496 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.211422    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"]
Sep 16 07:40:51.214773 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.214757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.217183 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.217164    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 07:40:51.217183 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.217180    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-27wd5\""
Sep 16 07:40:51.217350 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.217164    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 07:40:51.217690 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.217670    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 07:40:51.229360 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.229337    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"]
Sep 16 07:40:51.282721 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.282688    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfg6q\" (UniqueName: \"kubernetes.io/projected/4b1b26dd-431d-476c-9510-74a9f9369d03-kube-api-access-mfg6q\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.282906 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.282728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b1b26dd-431d-476c-9510-74a9f9369d03-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.282906 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.282750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b1b26dd-431d-476c-9510-74a9f9369d03-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.282906 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.282839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4b1b26dd-431d-476c-9510-74a9f9369d03-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.282906 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.282886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b1b26dd-431d-476c-9510-74a9f9369d03-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.283120 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.282923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1b26dd-431d-476c-9510-74a9f9369d03-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.283120 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.282993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1b26dd-431d-476c-9510-74a9f9369d03-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.384020 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.383981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b1b26dd-431d-476c-9510-74a9f9369d03-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.384020 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.384022    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b1b26dd-431d-476c-9510-74a9f9369d03-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.384220 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.384053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4b1b26dd-431d-476c-9510-74a9f9369d03-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.384298 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.384279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b1b26dd-431d-476c-9510-74a9f9369d03-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.384334 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.384319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1b26dd-431d-476c-9510-74a9f9369d03-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.384381 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.384344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1b26dd-431d-476c-9510-74a9f9369d03-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.384500 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.384420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfg6q\" (UniqueName: \"kubernetes.io/projected/4b1b26dd-431d-476c-9510-74a9f9369d03-kube-api-access-mfg6q\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.385067 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.385041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b1b26dd-431d-476c-9510-74a9f9369d03-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.386527 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.386498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1b26dd-431d-476c-9510-74a9f9369d03-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.386634 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.386572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b1b26dd-431d-476c-9510-74a9f9369d03-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.386888 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.386868    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b1b26dd-431d-476c-9510-74a9f9369d03-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.386933 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.386889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4b1b26dd-431d-476c-9510-74a9f9369d03-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.393231 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.393211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1b26dd-431d-476c-9510-74a9f9369d03-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.393611 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.393589    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfg6q\" (UniqueName: \"kubernetes.io/projected/4b1b26dd-431d-476c-9510-74a9f9369d03-kube-api-access-mfg6q\") pod \"istiod-openshift-gateway-5677ccb49b-qwxkf\" (UID: \"4b1b26dd-431d-476c-9510-74a9f9369d03\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.409789 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.409761    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:51.410741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.410722    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh"
Sep 16 07:40:51.523688 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.523598    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:51.800004 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:51.799970    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"]
Sep 16 07:40:51.802164 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:40:51.802131    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1b26dd_431d_476c_9510_74a9f9369d03.slice/crio-59d66175985c18afed6ef6e7ce5f0d571b60c20a25f9dd858adbe9d6fc056ada WatchSource:0}: Error finding container 59d66175985c18afed6ef6e7ce5f0d571b60c20a25f9dd858adbe9d6fc056ada: Status 404 returned error can't find the container with id 59d66175985c18afed6ef6e7ce5f0d571b60c20a25f9dd858adbe9d6fc056ada
Sep 16 07:40:52.414680 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:52.414645    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf" event={"ID":"4b1b26dd-431d-476c-9510-74a9f9369d03","Type":"ContainerStarted","Data":"59d66175985c18afed6ef6e7ce5f0d571b60c20a25f9dd858adbe9d6fc056ada"}
Sep 16 07:40:53.947249 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:53.947204    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xnnll"]
Sep 16 07:40:53.954727 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:53.954625    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xnnll"
Sep 16 07:40:53.963448 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:53.963058    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 16 07:40:53.963448 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:53.963206    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-g4r45\""
Sep 16 07:40:53.974932 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:53.974908    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xnnll"]
Sep 16 07:40:53.976328 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:53.976308    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8"]
Sep 16 07:40:53.979996 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:53.979977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8"
Sep 16 07:40:53.984265 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:53.984245    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 16 07:40:53.994301 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:53.994233    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-947p5\""
Sep 16 07:40:54.009177 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.009099    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8"]
Sep 16 07:40:54.112233 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.112192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qf9s\" (UniqueName: \"kubernetes.io/projected/5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb-kube-api-access-2qf9s\") pod \"kserve-controller-manager-d6f4b65b8-xnnll\" (UID: \"5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xnnll"
Sep 16 07:40:54.112410 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.112271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb-cert\") pod \"kserve-controller-manager-d6f4b65b8-xnnll\" (UID: \"5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xnnll"
Sep 16 07:40:54.112410 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.112296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/382d4153-d0b5-42e4-8232-5910ee40d6a4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-98nd8\" (UID: \"382d4153-d0b5-42e4-8232-5910ee40d6a4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8"
Sep 16 07:40:54.112410 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.112347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzp7k\" (UniqueName: \"kubernetes.io/projected/382d4153-d0b5-42e4-8232-5910ee40d6a4-kube-api-access-qzp7k\") pod \"llmisvc-controller-manager-777c6bd7d5-98nd8\" (UID: \"382d4153-d0b5-42e4-8232-5910ee40d6a4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8"
Sep 16 07:40:54.212910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.212799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzp7k\" (UniqueName: \"kubernetes.io/projected/382d4153-d0b5-42e4-8232-5910ee40d6a4-kube-api-access-qzp7k\") pod \"llmisvc-controller-manager-777c6bd7d5-98nd8\" (UID: \"382d4153-d0b5-42e4-8232-5910ee40d6a4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8"
Sep 16 07:40:54.212910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.212891    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qf9s\" (UniqueName: \"kubernetes.io/projected/5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb-kube-api-access-2qf9s\") pod \"kserve-controller-manager-d6f4b65b8-xnnll\" (UID: \"5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xnnll"
Sep 16 07:40:54.213131 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.212953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb-cert\") pod \"kserve-controller-manager-d6f4b65b8-xnnll\" (UID: \"5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xnnll"
Sep 16 07:40:54.213131 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.212976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/382d4153-d0b5-42e4-8232-5910ee40d6a4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-98nd8\" (UID: \"382d4153-d0b5-42e4-8232-5910ee40d6a4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8"
Sep 16 07:40:54.213131 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:40:54.213092    2581 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 07:40:54.213131 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:40:54.213108    2581 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 16 07:40:54.213342 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:40:54.213159    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/382d4153-d0b5-42e4-8232-5910ee40d6a4-cert podName:382d4153-d0b5-42e4-8232-5910ee40d6a4 nodeName:}" failed. No retries permitted until 2026-09-16 07:40:54.713138769 +0000 UTC m=+415.639750905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/382d4153-d0b5-42e4-8232-5910ee40d6a4-cert") pod "llmisvc-controller-manager-777c6bd7d5-98nd8" (UID: "382d4153-d0b5-42e4-8232-5910ee40d6a4") : secret "llmisvc-webhook-server-cert" not found
Sep 16 07:40:54.213342 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:40:54.213177    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb-cert podName:5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb nodeName:}" failed. No retries permitted until 2026-09-16 07:40:54.713168399 +0000 UTC m=+415.639780529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb-cert") pod "kserve-controller-manager-d6f4b65b8-xnnll" (UID: "5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb") : secret "kserve-webhook-server-cert" not found
Sep 16 07:40:54.225838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.225797    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qf9s\" (UniqueName: \"kubernetes.io/projected/5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb-kube-api-access-2qf9s\") pod \"kserve-controller-manager-d6f4b65b8-xnnll\" (UID: \"5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xnnll"
Sep 16 07:40:54.230070 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.230023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzp7k\" (UniqueName: \"kubernetes.io/projected/382d4153-d0b5-42e4-8232-5910ee40d6a4-kube-api-access-qzp7k\") pod \"llmisvc-controller-manager-777c6bd7d5-98nd8\" (UID: \"382d4153-d0b5-42e4-8232-5910ee40d6a4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8"
Sep 16 07:40:54.287670 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.287634    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:40:54.287783 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.287736    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:40:54.427112 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.427070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf" event={"ID":"4b1b26dd-431d-476c-9510-74a9f9369d03","Type":"ContainerStarted","Data":"c5ac72e45fec5df2d1e93550d822ca8513cd572f0db256ddbd79abc992ff6c35"}
Sep 16 07:40:54.427292 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.427238    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:54.467978 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.467880    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf" podStartSLOduration=0.984985642 podStartE2EDuration="3.467865769s" totalImagesPullingTime="2.482880127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:40:54.466007098 +0000 UTC m=+415.392619250" watchObservedRunningTime="2026-09-16 07:40:54.467865769 +0000 UTC m=+415.394477919"
Sep 16 07:40:54.717220 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.717184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb-cert\") pod \"kserve-controller-manager-d6f4b65b8-xnnll\" (UID: \"5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xnnll"
Sep 16 07:40:54.717220 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.717219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/382d4153-d0b5-42e4-8232-5910ee40d6a4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-98nd8\" (UID: \"382d4153-d0b5-42e4-8232-5910ee40d6a4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8"
Sep 16 07:40:54.719641 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.719573    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb-cert\") pod \"kserve-controller-manager-d6f4b65b8-xnnll\" (UID: \"5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xnnll"
Sep 16 07:40:54.719792 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.719770    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/382d4153-d0b5-42e4-8232-5910ee40d6a4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-98nd8\" (UID: \"382d4153-d0b5-42e4-8232-5910ee40d6a4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8"
Sep 16 07:40:54.872434 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.872397    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xnnll"
Sep 16 07:40:54.892489 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:54.892463    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8"
Sep 16 07:40:55.162074 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:55.162050    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xnnll"]
Sep 16 07:40:55.163773 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:40:55.163737    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd49635_f9ef_45f6_b9b9_7ab05df1ddeb.slice/crio-507cd3570eaf896e650e4cb35f1cb00a507c580547d3b7404df565e87d841a2c WatchSource:0}: Error finding container 507cd3570eaf896e650e4cb35f1cb00a507c580547d3b7404df565e87d841a2c: Status 404 returned error can't find the container with id 507cd3570eaf896e650e4cb35f1cb00a507c580547d3b7404df565e87d841a2c
Sep 16 07:40:55.180343 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:55.180320    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8"]
Sep 16 07:40:55.182330 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:40:55.182301    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod382d4153_d0b5_42e4_8232_5910ee40d6a4.slice/crio-941dea3e1536b3e8a0976009129fdd5c3a7e688774f3c2358c0767009ae68ea9 WatchSource:0}: Error finding container 941dea3e1536b3e8a0976009129fdd5c3a7e688774f3c2358c0767009ae68ea9: Status 404 returned error can't find the container with id 941dea3e1536b3e8a0976009129fdd5c3a7e688774f3c2358c0767009ae68ea9
Sep 16 07:40:55.432052 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:55.431964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xnnll" event={"ID":"5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb","Type":"ContainerStarted","Data":"507cd3570eaf896e650e4cb35f1cb00a507c580547d3b7404df565e87d841a2c"}
Sep 16 07:40:55.433099 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:55.433071    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8" event={"ID":"382d4153-d0b5-42e4-8232-5910ee40d6a4","Type":"ContainerStarted","Data":"941dea3e1536b3e8a0976009129fdd5c3a7e688774f3c2358c0767009ae68ea9"}
Sep 16 07:40:55.434814 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:55.434779    2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-qwxkf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 07:40:55.434925 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:55.434851    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf" podUID="4b1b26dd-431d-476c-9510-74a9f9369d03" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 07:40:58.434420 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:58.434392    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qwxkf"
Sep 16 07:40:58.449068 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:58.449033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xnnll" event={"ID":"5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb","Type":"ContainerStarted","Data":"e09e3b784617cd6fc4e5d52d244ff961aef04dab1ccd44c6cb91fbd3abc1bc19"}
Sep 16 07:40:58.449195 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:58.449096    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xnnll"
Sep 16 07:40:58.450486 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:58.450464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8" event={"ID":"382d4153-d0b5-42e4-8232-5910ee40d6a4","Type":"ContainerStarted","Data":"04a464c96c14b315b394a721fc52c0bb8623a5fcac551e2fb5a3f5cd354d3b43"}
Sep 16 07:40:58.450587 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:58.450576    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8"
Sep 16 07:40:58.480968 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:58.480925    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8" podStartSLOduration=2.649095875 podStartE2EDuration="5.480912542s" totalImagesPullingTime="2.831816667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:40:58.479990829 +0000 UTC m=+419.406602983" watchObservedRunningTime="2026-09-16 07:40:58.480912542 +0000 UTC m=+419.407524687"
Sep 16 07:40:58.501465 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:58.501410    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xnnll" podStartSLOduration=2.651326707 podStartE2EDuration="5.501388501s" totalImagesPullingTime="2.850061794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:40:58.500924298 +0000 UTC m=+419.427536452" watchObservedRunningTime="2026-09-16 07:40:58.501388501 +0000 UTC m=+419.428000856"
Sep 16 07:40:59.401996 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:40:59.401964    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-cnff7"
Sep 16 07:41:29.457478 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:41:29.457406    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-98nd8"
Sep 16 07:41:29.460563 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:41:29.460544    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xnnll"
Sep 16 07:42:16.243959 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:16.243918    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-vzns8"]
Sep 16 07:42:16.246072 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:16.246056    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-vzns8"
Sep 16 07:42:16.250024 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:16.250003    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 07:42:16.250263 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:16.250249    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 07:42:16.250648 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:16.250636    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-svhg6\""
Sep 16 07:42:16.275333 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:16.275306    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-vzns8"]
Sep 16 07:42:16.351177 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:16.351148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6t7z\" (UniqueName: \"kubernetes.io/projected/c1c72075-54a4-4918-b7de-b7c690cfb55d-kube-api-access-x6t7z\") pod \"authorino-operator-76b478bd78-vzns8\" (UID: \"c1c72075-54a4-4918-b7de-b7c690cfb55d\") " pod="kuadrant-system/authorino-operator-76b478bd78-vzns8"
Sep 16 07:42:16.452383 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:16.452358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6t7z\" (UniqueName: \"kubernetes.io/projected/c1c72075-54a4-4918-b7de-b7c690cfb55d-kube-api-access-x6t7z\") pod \"authorino-operator-76b478bd78-vzns8\" (UID: \"c1c72075-54a4-4918-b7de-b7c690cfb55d\") " pod="kuadrant-system/authorino-operator-76b478bd78-vzns8"
Sep 16 07:42:16.464785 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:16.464761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6t7z\" (UniqueName: \"kubernetes.io/projected/c1c72075-54a4-4918-b7de-b7c690cfb55d-kube-api-access-x6t7z\") pod \"authorino-operator-76b478bd78-vzns8\" (UID: \"c1c72075-54a4-4918-b7de-b7c690cfb55d\") " pod="kuadrant-system/authorino-operator-76b478bd78-vzns8"
Sep 16 07:42:16.555746 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:16.555690    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-vzns8"
Sep 16 07:42:16.823182 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:16.823158    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-vzns8"]
Sep 16 07:42:16.825790 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:42:16.825766    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c72075_54a4_4918_b7de_b7c690cfb55d.slice/crio-74646ce1b4ce45275c79dff0c74ec27818b06667860d6e184cdd45a40ec23412 WatchSource:0}: Error finding container 74646ce1b4ce45275c79dff0c74ec27818b06667860d6e184cdd45a40ec23412: Status 404 returned error can't find the container with id 74646ce1b4ce45275c79dff0c74ec27818b06667860d6e184cdd45a40ec23412
Sep 16 07:42:17.788299 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:17.788266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-vzns8" event={"ID":"c1c72075-54a4-4918-b7de-b7c690cfb55d","Type":"ContainerStarted","Data":"74646ce1b4ce45275c79dff0c74ec27818b06667860d6e184cdd45a40ec23412"}
Sep 16 07:42:27.831040 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:27.830964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-vzns8" event={"ID":"c1c72075-54a4-4918-b7de-b7c690cfb55d","Type":"ContainerStarted","Data":"0efed5e5f34f50230e15db3845ecb9a633aac3b000eeef9fbaf7fc96bf7a600a"}
Sep 16 07:42:27.831371 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:27.831094    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-vzns8"
Sep 16 07:42:27.853864 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:27.853794    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-vzns8" podStartSLOduration=1.106217826 podStartE2EDuration="11.853776314s" totalImagesPullingTime="10.747558488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:42:27.852924543 +0000 UTC m=+508.779536696" watchObservedRunningTime="2026-09-16 07:42:27.853776314 +0000 UTC m=+508.780388467"
Sep 16 07:42:38.838230 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:42:38.838197    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-vzns8"
Sep 16 07:43:04.429000 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.428914    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"]
Sep 16 07:43:04.432613 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.432588    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.435530 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.435510    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c6k94\""
Sep 16 07:43:04.444438 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.444412    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"]
Sep 16 07:43:04.522734 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.522703    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.522895 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.522749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.522895 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.522771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.522895 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.522859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.522895 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.522882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.523063 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.522915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.523063 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.522948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.523063 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.522963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.523063 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.522978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbb67\" (UniqueName: \"kubernetes.io/projected/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-kube-api-access-bbb67\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.623944 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.623912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.624099 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.623968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.624099 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.623996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.624099 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.624020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbb67\" (UniqueName: \"kubernetes.io/projected/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-kube-api-access-bbb67\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.624099 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.624054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.624297 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.624190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.624297 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.624237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.624297 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.624280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.624451 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.624308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.624451 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.624334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.624451 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.624382    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.624842 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.624788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.624842 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.624788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.624998 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.624861    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.626653 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.626634    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.626736 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.626677    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.631977 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.631945    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.632058 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.631978    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbb67\" (UniqueName: \"kubernetes.io/projected/7c1ff10d-6e1c-4838-a5c6-ccd8e4459859-kube-api-access-bbb67\") pod \"maas-default-gateway-openshift-default-78bb78ff78-j494b\" (UID: \"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:04.746150 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:04.746074    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:05.026288 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:05.026191    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"]
Sep 16 07:43:05.029383 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:43:05.029351    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1ff10d_6e1c_4838_a5c6_ccd8e4459859.slice/crio-94276b754abedc016222522553bf690387c8e39b7959497cf46814a3162fb2bc WatchSource:0}: Error finding container 94276b754abedc016222522553bf690387c8e39b7959497cf46814a3162fb2bc: Status 404 returned error can't find the container with id 94276b754abedc016222522553bf690387c8e39b7959497cf46814a3162fb2bc
Sep 16 07:43:05.031347 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:05.031305    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:43:05.031461 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:05.031390    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:43:05.031461 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:05.031432    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:43:05.989124 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:05.989092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b" event={"ID":"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859","Type":"ContainerStarted","Data":"f1c23886e323aca4e635c859fc22857409ea13a5df4301f4c3d197fa3bdb919c"}
Sep 16 07:43:05.989484 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:05.989150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b" event={"ID":"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859","Type":"ContainerStarted","Data":"94276b754abedc016222522553bf690387c8e39b7959497cf46814a3162fb2bc"}
Sep 16 07:43:06.008520 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:06.008469    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b" podStartSLOduration=2.008454325 podStartE2EDuration="2.008454325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:43:06.00749696 +0000 UTC m=+546.934109114" watchObservedRunningTime="2026-09-16 07:43:06.008454325 +0000 UTC m=+546.935066478"
Sep 16 07:43:06.746229 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:06.746195    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:06.750889 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:06.750863    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:06.993429 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:06.993401    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:06.994193 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:06.994176    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:43:43.033095 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.033057    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65f5558984-b2lrm"]
Sep 16 07:43:43.036844 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.036813    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5558984-b2lrm"
Sep 16 07:43:43.039363 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.039339    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 07:43:43.039491 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.039339    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 07:43:43.039491 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.039378    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 07:43:43.039491 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.039478    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nvkmk\""
Sep 16 07:43:43.043806 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.043781    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f5558984-b2lrm"]
Sep 16 07:43:43.156520 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.156490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d75330ad-3aeb-4725-b04b-1dd931e3abb8-webhook-cert\") pod \"maas-controller-65f5558984-b2lrm\" (UID: \"d75330ad-3aeb-4725-b04b-1dd931e3abb8\") " pod="opendatahub/maas-controller-65f5558984-b2lrm"
Sep 16 07:43:43.156692 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.156545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d75330ad-3aeb-4725-b04b-1dd931e3abb8-metrics-certs\") pod \"maas-controller-65f5558984-b2lrm\" (UID: \"d75330ad-3aeb-4725-b04b-1dd931e3abb8\") " pod="opendatahub/maas-controller-65f5558984-b2lrm"
Sep 16 07:43:43.156692 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.156672    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj5b8\" (UniqueName: \"kubernetes.io/projected/d75330ad-3aeb-4725-b04b-1dd931e3abb8-kube-api-access-jj5b8\") pod \"maas-controller-65f5558984-b2lrm\" (UID: \"d75330ad-3aeb-4725-b04b-1dd931e3abb8\") " pod="opendatahub/maas-controller-65f5558984-b2lrm"
Sep 16 07:43:43.258089 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.258050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj5b8\" (UniqueName: \"kubernetes.io/projected/d75330ad-3aeb-4725-b04b-1dd931e3abb8-kube-api-access-jj5b8\") pod \"maas-controller-65f5558984-b2lrm\" (UID: \"d75330ad-3aeb-4725-b04b-1dd931e3abb8\") " pod="opendatahub/maas-controller-65f5558984-b2lrm"
Sep 16 07:43:43.258270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.258137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d75330ad-3aeb-4725-b04b-1dd931e3abb8-webhook-cert\") pod \"maas-controller-65f5558984-b2lrm\" (UID: \"d75330ad-3aeb-4725-b04b-1dd931e3abb8\") " pod="opendatahub/maas-controller-65f5558984-b2lrm"
Sep 16 07:43:43.258270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.258207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d75330ad-3aeb-4725-b04b-1dd931e3abb8-metrics-certs\") pod \"maas-controller-65f5558984-b2lrm\" (UID: \"d75330ad-3aeb-4725-b04b-1dd931e3abb8\") " pod="opendatahub/maas-controller-65f5558984-b2lrm"
Sep 16 07:43:43.260888 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.260861    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d75330ad-3aeb-4725-b04b-1dd931e3abb8-webhook-cert\") pod \"maas-controller-65f5558984-b2lrm\" (UID: \"d75330ad-3aeb-4725-b04b-1dd931e3abb8\") " pod="opendatahub/maas-controller-65f5558984-b2lrm"
Sep 16 07:43:43.261015 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.260910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d75330ad-3aeb-4725-b04b-1dd931e3abb8-metrics-certs\") pod \"maas-controller-65f5558984-b2lrm\" (UID: \"d75330ad-3aeb-4725-b04b-1dd931e3abb8\") " pod="opendatahub/maas-controller-65f5558984-b2lrm"
Sep 16 07:43:43.267746 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.267726    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj5b8\" (UniqueName: \"kubernetes.io/projected/d75330ad-3aeb-4725-b04b-1dd931e3abb8-kube-api-access-jj5b8\") pod \"maas-controller-65f5558984-b2lrm\" (UID: \"d75330ad-3aeb-4725-b04b-1dd931e3abb8\") " pod="opendatahub/maas-controller-65f5558984-b2lrm"
Sep 16 07:43:43.349713 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.349662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5558984-b2lrm"
Sep 16 07:43:43.616719 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:43.616698    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f5558984-b2lrm"]
Sep 16 07:43:43.618919 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:43:43.618894    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75330ad_3aeb_4725_b04b_1dd931e3abb8.slice/crio-19620a7bf7276eb98be74ee8976deb2bb74b4a25602466e6dd3ad6b18c2b4703 WatchSource:0}: Error finding container 19620a7bf7276eb98be74ee8976deb2bb74b4a25602466e6dd3ad6b18c2b4703: Status 404 returned error can't find the container with id 19620a7bf7276eb98be74ee8976deb2bb74b4a25602466e6dd3ad6b18c2b4703
Sep 16 07:43:44.154403 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:44.154367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5558984-b2lrm" event={"ID":"d75330ad-3aeb-4725-b04b-1dd931e3abb8","Type":"ContainerStarted","Data":"19620a7bf7276eb98be74ee8976deb2bb74b4a25602466e6dd3ad6b18c2b4703"}
Sep 16 07:43:46.165662 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:46.165616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5558984-b2lrm" event={"ID":"d75330ad-3aeb-4725-b04b-1dd931e3abb8","Type":"ContainerStarted","Data":"0198b479cf88eeefc41f2acd7504aab54028dedbdb07efdd0ea819cb9ac27e0c"}
Sep 16 07:43:46.166091 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:46.165765    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65f5558984-b2lrm"
Sep 16 07:43:57.173118 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:57.173085    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65f5558984-b2lrm"
Sep 16 07:43:57.192354 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:57.192303    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65f5558984-b2lrm" podStartSLOduration=11.89403377 podStartE2EDuration="14.192287764s" totalImagesPullingTime="2.298253994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:43:46.19641023 +0000 UTC m=+587.123022383" watchObservedRunningTime="2026-09-16 07:43:57.192287764 +0000 UTC m=+598.118899917"
Sep 16 07:43:57.661621 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:57.661591    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f5558984-b2lrm"]
Sep 16 07:43:57.661772 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:57.661756    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-65f5558984-b2lrm" podUID="d75330ad-3aeb-4725-b04b-1dd931e3abb8" containerName="manager" containerID="cri-o://0198b479cf88eeefc41f2acd7504aab54028dedbdb07efdd0ea819cb9ac27e0c" gracePeriod=30
Sep 16 07:43:57.908539 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:57.908514    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5558984-b2lrm"
Sep 16 07:43:57.968387 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:57.968325    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj5b8\" (UniqueName: \"kubernetes.io/projected/d75330ad-3aeb-4725-b04b-1dd931e3abb8-kube-api-access-jj5b8\") pod \"d75330ad-3aeb-4725-b04b-1dd931e3abb8\" (UID: \"d75330ad-3aeb-4725-b04b-1dd931e3abb8\") "
Sep 16 07:43:57.968387 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:57.968381    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d75330ad-3aeb-4725-b04b-1dd931e3abb8-webhook-cert\") pod \"d75330ad-3aeb-4725-b04b-1dd931e3abb8\" (UID: \"d75330ad-3aeb-4725-b04b-1dd931e3abb8\") "
Sep 16 07:43:57.968579 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:57.968428    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d75330ad-3aeb-4725-b04b-1dd931e3abb8-metrics-certs\") pod \"d75330ad-3aeb-4725-b04b-1dd931e3abb8\" (UID: \"d75330ad-3aeb-4725-b04b-1dd931e3abb8\") "
Sep 16 07:43:57.970233 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:57.970205    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75330ad-3aeb-4725-b04b-1dd931e3abb8-kube-api-access-jj5b8" (OuterVolumeSpecName: "kube-api-access-jj5b8") pod "d75330ad-3aeb-4725-b04b-1dd931e3abb8" (UID: "d75330ad-3aeb-4725-b04b-1dd931e3abb8"). InnerVolumeSpecName "kube-api-access-jj5b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:43:57.970333 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:57.970268    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d75330ad-3aeb-4725-b04b-1dd931e3abb8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d75330ad-3aeb-4725-b04b-1dd931e3abb8" (UID: "d75330ad-3aeb-4725-b04b-1dd931e3abb8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 07:43:57.970389 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:57.970346    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d75330ad-3aeb-4725-b04b-1dd931e3abb8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d75330ad-3aeb-4725-b04b-1dd931e3abb8" (UID: "d75330ad-3aeb-4725-b04b-1dd931e3abb8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 07:43:58.069039 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:58.069009    2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d75330ad-3aeb-4725-b04b-1dd931e3abb8-webhook-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:43:58.069039 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:58.069032    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d75330ad-3aeb-4725-b04b-1dd931e3abb8-metrics-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:43:58.069039 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:58.069040    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj5b8\" (UniqueName: \"kubernetes.io/projected/d75330ad-3aeb-4725-b04b-1dd931e3abb8-kube-api-access-jj5b8\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:43:58.218028 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:58.217994    2581 generic.go:356] "Generic (PLEG): container finished" podID="d75330ad-3aeb-4725-b04b-1dd931e3abb8" containerID="0198b479cf88eeefc41f2acd7504aab54028dedbdb07efdd0ea819cb9ac27e0c" exitCode=0
Sep 16 07:43:58.218386 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:58.218044    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5558984-b2lrm"
Sep 16 07:43:58.218386 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:58.218067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5558984-b2lrm" event={"ID":"d75330ad-3aeb-4725-b04b-1dd931e3abb8","Type":"ContainerDied","Data":"0198b479cf88eeefc41f2acd7504aab54028dedbdb07efdd0ea819cb9ac27e0c"}
Sep 16 07:43:58.218386 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:58.218116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5558984-b2lrm" event={"ID":"d75330ad-3aeb-4725-b04b-1dd931e3abb8","Type":"ContainerDied","Data":"19620a7bf7276eb98be74ee8976deb2bb74b4a25602466e6dd3ad6b18c2b4703"}
Sep 16 07:43:58.218386 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:58.218147    2581 scope.go:117] "RemoveContainer" containerID="0198b479cf88eeefc41f2acd7504aab54028dedbdb07efdd0ea819cb9ac27e0c"
Sep 16 07:43:58.226615 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:58.226595    2581 scope.go:117] "RemoveContainer" containerID="0198b479cf88eeefc41f2acd7504aab54028dedbdb07efdd0ea819cb9ac27e0c"
Sep 16 07:43:58.226866 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:43:58.226844    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0198b479cf88eeefc41f2acd7504aab54028dedbdb07efdd0ea819cb9ac27e0c\": container with ID starting with 0198b479cf88eeefc41f2acd7504aab54028dedbdb07efdd0ea819cb9ac27e0c not found: ID does not exist" containerID="0198b479cf88eeefc41f2acd7504aab54028dedbdb07efdd0ea819cb9ac27e0c"
Sep 16 07:43:58.226937 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:58.226879    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0198b479cf88eeefc41f2acd7504aab54028dedbdb07efdd0ea819cb9ac27e0c"} err="failed to get container status \"0198b479cf88eeefc41f2acd7504aab54028dedbdb07efdd0ea819cb9ac27e0c\": rpc error: code = NotFound desc = could not find container \"0198b479cf88eeefc41f2acd7504aab54028dedbdb07efdd0ea819cb9ac27e0c\": container with ID starting with 0198b479cf88eeefc41f2acd7504aab54028dedbdb07efdd0ea819cb9ac27e0c not found: ID does not exist"
Sep 16 07:43:58.238736 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:58.238714    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f5558984-b2lrm"]
Sep 16 07:43:58.242237 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:58.242215    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65f5558984-b2lrm"]
Sep 16 07:43:59.504559 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:59.504535    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 07:43:59.504996 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:59.504538    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 07:43:59.683153 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:43:59.683123    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d75330ad-3aeb-4725-b04b-1dd931e3abb8" path="/var/lib/kubelet/pods/d75330ad-3aeb-4725-b04b-1dd931e3abb8/volumes"
Sep 16 07:44:19.992661 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:19.992629    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j"]
Sep 16 07:44:19.993125 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:19.992976    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d75330ad-3aeb-4725-b04b-1dd931e3abb8" containerName="manager"
Sep 16 07:44:19.993125 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:19.992989    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d75330ad-3aeb-4725-b04b-1dd931e3abb8" containerName="manager"
Sep 16 07:44:19.993125 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:19.993039    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d75330ad-3aeb-4725-b04b-1dd931e3abb8" containerName="manager"
Sep 16 07:44:19.997876 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:19.997853    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j"
Sep 16 07:44:20.000482 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:20.000454    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jfvmp\""
Sep 16 07:44:20.000629 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:20.000610    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 07:44:20.008801 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:20.008777    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j"]
Sep 16 07:44:20.050799 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:20.050776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92b9k\" (UniqueName: \"kubernetes.io/projected/dd4dc9bf-a10b-40fe-8136-9ba3b40549d3-kube-api-access-92b9k\") pod \"payload-pre-processing-59fbc47bf8-svx9j\" (UID: \"dd4dc9bf-a10b-40fe-8136-9ba3b40549d3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j"
Sep 16 07:44:20.050919 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:20.050878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd4dc9bf-a10b-40fe-8136-9ba3b40549d3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-svx9j\" (UID: \"dd4dc9bf-a10b-40fe-8136-9ba3b40549d3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j"
Sep 16 07:44:20.151746 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:20.151707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd4dc9bf-a10b-40fe-8136-9ba3b40549d3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-svx9j\" (UID: \"dd4dc9bf-a10b-40fe-8136-9ba3b40549d3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j"
Sep 16 07:44:20.151931 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:20.151773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92b9k\" (UniqueName: \"kubernetes.io/projected/dd4dc9bf-a10b-40fe-8136-9ba3b40549d3-kube-api-access-92b9k\") pod \"payload-pre-processing-59fbc47bf8-svx9j\" (UID: \"dd4dc9bf-a10b-40fe-8136-9ba3b40549d3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j"
Sep 16 07:44:20.152482 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:20.152461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd4dc9bf-a10b-40fe-8136-9ba3b40549d3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-svx9j\" (UID: \"dd4dc9bf-a10b-40fe-8136-9ba3b40549d3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j"
Sep 16 07:44:20.160146 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:20.160123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92b9k\" (UniqueName: \"kubernetes.io/projected/dd4dc9bf-a10b-40fe-8136-9ba3b40549d3-kube-api-access-92b9k\") pod \"payload-pre-processing-59fbc47bf8-svx9j\" (UID: \"dd4dc9bf-a10b-40fe-8136-9ba3b40549d3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j"
Sep 16 07:44:20.308904 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:20.308838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j"
Sep 16 07:44:20.575550 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:20.575525    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j"]
Sep 16 07:44:20.577196 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:44:20.577161    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4dc9bf_a10b_40fe_8136_9ba3b40549d3.slice/crio-c094c42f46e0a2e02c8b9287be2f300c74b2e02e4c48dd5ab0ce239fa3b66bf5 WatchSource:0}: Error finding container c094c42f46e0a2e02c8b9287be2f300c74b2e02e4c48dd5ab0ce239fa3b66bf5: Status 404 returned error can't find the container with id c094c42f46e0a2e02c8b9287be2f300c74b2e02e4c48dd5ab0ce239fa3b66bf5
Sep 16 07:44:21.341322 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:21.341280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j" event={"ID":"dd4dc9bf-a10b-40fe-8136-9ba3b40549d3","Type":"ContainerStarted","Data":"c094c42f46e0a2e02c8b9287be2f300c74b2e02e4c48dd5ab0ce239fa3b66bf5"}
Sep 16 07:44:22.347073 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:22.347036    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j" event={"ID":"dd4dc9bf-a10b-40fe-8136-9ba3b40549d3","Type":"ContainerStarted","Data":"e78edf7b271d58ad30ca5235f4a4ce56d59681b0dedc638a069fe2a03dbb905a"}
Sep 16 07:44:22.347439 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:22.347142    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j"
Sep 16 07:44:22.363267 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:22.363174    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j" podStartSLOduration=1.8414322300000001 podStartE2EDuration="3.363159341s" totalImagesPullingTime="1.521727111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:44:22.36220972 +0000 UTC m=+623.288821890" watchObservedRunningTime="2026-09-16 07:44:22.363159341 +0000 UTC m=+623.289771540"
Sep 16 07:44:29.388755 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:29.388675    2581 generic.go:356] "Generic (PLEG): container finished" podID="7c1ff10d-6e1c-4838-a5c6-ccd8e4459859" containerID="f1c23886e323aca4e635c859fc22857409ea13a5df4301f4c3d197fa3bdb919c" exitCode=0
Sep 16 07:44:29.389168 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:29.388751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b" event={"ID":"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859","Type":"ContainerDied","Data":"f1c23886e323aca4e635c859fc22857409ea13a5df4301f4c3d197fa3bdb919c"}
Sep 16 07:44:29.389168 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:29.389165    2581 scope.go:117] "RemoveContainer" containerID="f1c23886e323aca4e635c859fc22857409ea13a5df4301f4c3d197fa3bdb919c"
Sep 16 07:44:29.390156 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:29.390131    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:44:29.390220 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:29.390197    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:44:29.390255 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:29.390229    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:44:30.394974 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:30.394943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b" event={"ID":"7c1ff10d-6e1c-4838-a5c6-ccd8e4459859","Type":"ContainerStarted","Data":"b6f0a6f556a3fd8d0d511a2139bf6fda3f73b7dad8be9cc3b3271b53e6b49705"}
Sep 16 07:44:30.747113 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:30.747078    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:44:30.747324 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:30.747299    2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-j494b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 16 07:44:30.747370 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:30.747355    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b" podUID="7c1ff10d-6e1c-4838-a5c6-ccd8e4459859" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 16 07:44:31.747168 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:31.747134    2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-j494b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 16 07:44:31.747538 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:31.747185    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b" podUID="7c1ff10d-6e1c-4838-a5c6-ccd8e4459859" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 16 07:44:32.747181 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:32.747149    2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-j494b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 16 07:44:32.747547 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:32.747207    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b" podUID="7c1ff10d-6e1c-4838-a5c6-ccd8e4459859" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 16 07:44:33.353997 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:33.353967    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-svx9j"
Sep 16 07:44:33.747008 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:33.746975    2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-j494b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 16 07:44:33.747176 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:33.747030    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b" podUID="7c1ff10d-6e1c-4838-a5c6-ccd8e4459859" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 16 07:44:34.747028 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:34.746991    2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-j494b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Sep 16 07:44:34.747413 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:34.747049    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b" podUID="7c1ff10d-6e1c-4838-a5c6-ccd8e4459859" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Sep 16 07:44:35.750835 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:35.750793    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:44:35.751210 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:35.751063    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:44:35.751647 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:35.751630    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-j494b"
Sep 16 07:44:41.163986 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.163956    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"]
Sep 16 07:44:41.171916 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.171882    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.175420 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.175392    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 07:44:41.175550 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.175453    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 07:44:41.175550 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.175472    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t7r7m\""
Sep 16 07:44:41.175666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.175396    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 16 07:44:41.176716 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.176679    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"]
Sep 16 07:44:41.233765 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.233742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bbedf33-afb5-4a29-b905-35d1a9ea4023-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.233906 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.233773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8hsb\" (UniqueName: \"kubernetes.io/projected/1bbedf33-afb5-4a29-b905-35d1a9ea4023-kube-api-access-b8hsb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.233906 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.233792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bbedf33-afb5-4a29-b905-35d1a9ea4023-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.233906 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.233832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bbedf33-afb5-4a29-b905-35d1a9ea4023-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.234064 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.233950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bbedf33-afb5-4a29-b905-35d1a9ea4023-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.234064 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.233997    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bbedf33-afb5-4a29-b905-35d1a9ea4023-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.334705 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.334672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bbedf33-afb5-4a29-b905-35d1a9ea4023-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.334888 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.334713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8hsb\" (UniqueName: \"kubernetes.io/projected/1bbedf33-afb5-4a29-b905-35d1a9ea4023-kube-api-access-b8hsb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.334888 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.334740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bbedf33-afb5-4a29-b905-35d1a9ea4023-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.334888 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.334762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bbedf33-afb5-4a29-b905-35d1a9ea4023-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.334888 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.334856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bbedf33-afb5-4a29-b905-35d1a9ea4023-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.335099 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.334889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bbedf33-afb5-4a29-b905-35d1a9ea4023-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.335212 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.335153    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bbedf33-afb5-4a29-b905-35d1a9ea4023-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.335271 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.335257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bbedf33-afb5-4a29-b905-35d1a9ea4023-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.335354 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.335333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bbedf33-afb5-4a29-b905-35d1a9ea4023-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.337088 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.337069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bbedf33-afb5-4a29-b905-35d1a9ea4023-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.337232 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.337216    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bbedf33-afb5-4a29-b905-35d1a9ea4023-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.343066 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.343041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8hsb\" (UniqueName: \"kubernetes.io/projected/1bbedf33-afb5-4a29-b905-35d1a9ea4023-kube-api-access-b8hsb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4\" (UID: \"1bbedf33-afb5-4a29-b905-35d1a9ea4023\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.484890 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.484810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:41.749723 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.749700    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"]
Sep 16 07:44:41.752019 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:44:41.751985    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bbedf33_afb5_4a29_b905_35d1a9ea4023.slice/crio-ad707f0b8820102dc98a76e20fe45c55d231edfe6e9c2f02b9941438c52b5a95 WatchSource:0}: Error finding container ad707f0b8820102dc98a76e20fe45c55d231edfe6e9c2f02b9941438c52b5a95: Status 404 returned error can't find the container with id ad707f0b8820102dc98a76e20fe45c55d231edfe6e9c2f02b9941438c52b5a95
Sep 16 07:44:41.753425 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:41.753404    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 07:44:42.452961 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:42.452928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4" event={"ID":"1bbedf33-afb5-4a29-b905-35d1a9ea4023","Type":"ContainerStarted","Data":"ad707f0b8820102dc98a76e20fe45c55d231edfe6e9c2f02b9941438c52b5a95"}
Sep 16 07:44:43.460088 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:43.460052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4" event={"ID":"1bbedf33-afb5-4a29-b905-35d1a9ea4023","Type":"ContainerStarted","Data":"be997575492ff5a6994b7905f929aec687442430e0a2919bf5fcda3e7c813dab"}
Sep 16 07:44:43.477437 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:43.477388    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4" podStartSLOduration=1.378530689 podStartE2EDuration="2.477373136s" totalImagesPullingTime="1.098842447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:44:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:44:43.475611261 +0000 UTC m=+644.402223413" watchObservedRunningTime="2026-09-16 07:44:43.477373136 +0000 UTC m=+644.403985289"
Sep 16 07:44:51.485944 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.485913    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:51.486459 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.485972    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:51.498309 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.498281    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:51.509643 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.509612    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4"
Sep 16 07:44:51.568580 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.568550    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"]
Sep 16 07:44:51.571163 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.571139    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.573436 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.573416    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 16 07:44:51.582975 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.582954    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"]
Sep 16 07:44:51.724427 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.724400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c8f434-9910-4015-abac-e9cd82c1f223-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.724427 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.724429    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snbmr\" (UniqueName: \"kubernetes.io/projected/12c8f434-9910-4015-abac-e9cd82c1f223-kube-api-access-snbmr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.724661 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.724456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c8f434-9910-4015-abac-e9cd82c1f223-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.724661 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.724488    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c8f434-9910-4015-abac-e9cd82c1f223-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.724661 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.724525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c8f434-9910-4015-abac-e9cd82c1f223-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.724661 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.724553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c8f434-9910-4015-abac-e9cd82c1f223-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.825545 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.825460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c8f434-9910-4015-abac-e9cd82c1f223-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.825708 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.825574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snbmr\" (UniqueName: \"kubernetes.io/projected/12c8f434-9910-4015-abac-e9cd82c1f223-kube-api-access-snbmr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.825708 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.825617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c8f434-9910-4015-abac-e9cd82c1f223-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.825708 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.825652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c8f434-9910-4015-abac-e9cd82c1f223-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.825708 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.825693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c8f434-9910-4015-abac-e9cd82c1f223-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.825942 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.825737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c8f434-9910-4015-abac-e9cd82c1f223-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.826159 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.826136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c8f434-9910-4015-abac-e9cd82c1f223-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.826501 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.826397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c8f434-9910-4015-abac-e9cd82c1f223-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.826501 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.826482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c8f434-9910-4015-abac-e9cd82c1f223-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.828650 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.828624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c8f434-9910-4015-abac-e9cd82c1f223-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.829000 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.828979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c8f434-9910-4015-abac-e9cd82c1f223-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.834929 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.834901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snbmr\" (UniqueName: \"kubernetes.io/projected/12c8f434-9910-4015-abac-e9cd82c1f223-kube-api-access-snbmr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg\" (UID: \"12c8f434-9910-4015-abac-e9cd82c1f223\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:51.882179 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:51.882152    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:44:52.154689 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:52.154662    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"]
Sep 16 07:44:52.156596 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:44:52.156566    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c8f434_9910_4015_abac_e9cd82c1f223.slice/crio-4db3c35db2bfe1f7c0a05329353104d73c463cb06e643b612c01bbf931fee3a2 WatchSource:0}: Error finding container 4db3c35db2bfe1f7c0a05329353104d73c463cb06e643b612c01bbf931fee3a2: Status 404 returned error can't find the container with id 4db3c35db2bfe1f7c0a05329353104d73c463cb06e643b612c01bbf931fee3a2
Sep 16 07:44:52.502206 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:52.502109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg" event={"ID":"12c8f434-9910-4015-abac-e9cd82c1f223","Type":"ContainerStarted","Data":"9dc203313e5f6081e54d18edd91e8d93440cc94eb0bcb96d1caf1df93faa5bfc"}
Sep 16 07:44:52.502206 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:52.502161    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg" event={"ID":"12c8f434-9910-4015-abac-e9cd82c1f223","Type":"ContainerStarted","Data":"4db3c35db2bfe1f7c0a05329353104d73c463cb06e643b612c01bbf931fee3a2"}
Sep 16 07:44:52.527156 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:52.527102    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg" podStartSLOduration=1.358314268 podStartE2EDuration="1.527085837s" totalImagesPullingTime="168.771569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:44:52.524613535 +0000 UTC m=+653.451225688" watchObservedRunningTime="2026-09-16 07:44:52.527085837 +0000 UTC m=+653.453697989"
Sep 16 07:44:59.549590 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.549555    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"]
Sep 16 07:44:59.551934 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.551913    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.554205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.554183    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 16 07:44:59.565065 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.565044    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"]
Sep 16 07:44:59.690574 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.690541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352a18d0-13ab-4b27-aecf-98506046e069-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.690718 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.690583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352a18d0-13ab-4b27-aecf-98506046e069-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.690718 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.690604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352a18d0-13ab-4b27-aecf-98506046e069-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.690718 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.690643    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr589\" (UniqueName: \"kubernetes.io/projected/352a18d0-13ab-4b27-aecf-98506046e069-kube-api-access-qr589\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.690718 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.690695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352a18d0-13ab-4b27-aecf-98506046e069-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.690879 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.690742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352a18d0-13ab-4b27-aecf-98506046e069-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.791777 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.791743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352a18d0-13ab-4b27-aecf-98506046e069-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.791777 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.791777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr589\" (UniqueName: \"kubernetes.io/projected/352a18d0-13ab-4b27-aecf-98506046e069-kube-api-access-qr589\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.792044 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.791838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352a18d0-13ab-4b27-aecf-98506046e069-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.792243 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.792215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352a18d0-13ab-4b27-aecf-98506046e069-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.792385 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.792279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/352a18d0-13ab-4b27-aecf-98506046e069-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.792385 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.792347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352a18d0-13ab-4b27-aecf-98506046e069-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.792553 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.792395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352a18d0-13ab-4b27-aecf-98506046e069-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.792627 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.792602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/352a18d0-13ab-4b27-aecf-98506046e069-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.792699 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.792685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/352a18d0-13ab-4b27-aecf-98506046e069-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.794121 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.794102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/352a18d0-13ab-4b27-aecf-98506046e069-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.794519 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.794497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/352a18d0-13ab-4b27-aecf-98506046e069-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.799646 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.799597    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr589\" (UniqueName: \"kubernetes.io/projected/352a18d0-13ab-4b27-aecf-98506046e069-kube-api-access-qr589\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv4qz\" (UID: \"352a18d0-13ab-4b27-aecf-98506046e069\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:44:59.862996 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:44:59.862971    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:45:00.135845 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:45:00.135803    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"]
Sep 16 07:45:00.138460 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:45:00.138430    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352a18d0_13ab_4b27_aecf_98506046e069.slice/crio-663f96d1f24370592d18fd9c8ca1568067cd452f6bbb6aa877b9b8336ee5b4ea WatchSource:0}: Error finding container 663f96d1f24370592d18fd9c8ca1568067cd452f6bbb6aa877b9b8336ee5b4ea: Status 404 returned error can't find the container with id 663f96d1f24370592d18fd9c8ca1568067cd452f6bbb6aa877b9b8336ee5b4ea
Sep 16 07:45:00.539549 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:45:00.539519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz" event={"ID":"352a18d0-13ab-4b27-aecf-98506046e069","Type":"ContainerStarted","Data":"c42bbd906d4c55d64cf830266ce7305e9aaee92a353b4aa75daf3b536b348bbe"}
Sep 16 07:45:00.539710 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:45:00.539566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz" event={"ID":"352a18d0-13ab-4b27-aecf-98506046e069","Type":"ContainerStarted","Data":"663f96d1f24370592d18fd9c8ca1568067cd452f6bbb6aa877b9b8336ee5b4ea"}
Sep 16 07:45:00.558025 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:45:00.557979    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz" podStartSLOduration=1.369257137 podStartE2EDuration="1.557964456s" totalImagesPullingTime="188.707319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:45:00.556297468 +0000 UTC m=+661.482909621" watchObservedRunningTime="2026-09-16 07:45:00.557964456 +0000 UTC m=+661.484576609"
Sep 16 07:45:01.882590 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:45:01.882551    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:45:01.883092 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:45:01.882649    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:45:01.895393 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:45:01.895370    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:45:02.563812 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:45:02.563782    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg"
Sep 16 07:45:09.864164 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:45:09.864139    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:45:09.864518 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:45:09.864190    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:45:09.876458 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:45:09.876435    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:45:10.599452 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:45:10.599425    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv4qz"
Sep 16 07:47:00.539898 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.539862    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"]
Sep 16 07:47:00.542709 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.542688    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.545694 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.545674    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c9020d-openshift-default-dockercfg-fcjzp\""
Sep 16 07:47:00.556471 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.556444    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"]
Sep 16 07:47:00.597056 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.597019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vn99\" (UniqueName: \"kubernetes.io/projected/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-kube-api-access-4vn99\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.597333 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.597312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-workload-socket\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.597485 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.597467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-envoy\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.597627 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.597610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-podinfo\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.597733 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.597713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istiod-ca-cert\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.597814 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.597772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-credential-socket\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.597814 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.597799    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-data\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.597942 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.597851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-workload-certs\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.597942 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.597920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-token\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.617333 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.617297    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"]
Sep 16 07:47:00.619985 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.619953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.622657 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.622635    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-fc27a5-openshift-default-dockercfg-29s5m\""
Sep 16 07:47:00.632475 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.632453    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"]
Sep 16 07:47:00.691497 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.691458    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"]
Sep 16 07:47:00.699675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.698588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8999f450-ddca-4885-af15-c203abdf47d7-istiod-ca-cert\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.699675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.698672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istiod-ca-cert\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.699675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.698700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-istio-data\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.699675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.698728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-istio-envoy\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.699675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.698770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-credential-socket\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.699675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.698841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-data\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.699675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.698872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-workload-certs\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.699675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.698970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-token\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.699675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.698996    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-credential-socket\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.699675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.699024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8999f450-ddca-4885-af15-c203abdf47d7-istio-token\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.699675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.699049    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz4wf\" (UniqueName: \"kubernetes.io/projected/8999f450-ddca-4885-af15-c203abdf47d7-kube-api-access-sz4wf\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.699675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.699090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-workload-socket\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.699675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.699127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-workload-certs\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.699675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.699173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vn99\" (UniqueName: \"kubernetes.io/projected/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-kube-api-access-4vn99\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.700532 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.699211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-workload-socket\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.700532 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.699235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-envoy\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.700532 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.699261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-podinfo\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.700532 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.699291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8999f450-ddca-4885-af15-c203abdf47d7-istio-podinfo\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.700532 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.699958    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-credential-socket\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.700532 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.700057    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-data\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.700532 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.700454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istiod-ca-cert\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.700532 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.700497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-workload-certs\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.700952 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.700687    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-workload-socket\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.702791 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.702752    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-envoy\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.703328 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.703309    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-podinfo\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.703411 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.703381    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.705735 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.705710    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5dd30f-openshift-default-dockercfg-5jqzw\""
Sep 16 07:47:00.714153 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.713362    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vn99\" (UniqueName: \"kubernetes.io/projected/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-kube-api-access-4vn99\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.714153 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.713864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-token\") pod \"e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") " pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.714153 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.714069    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"]
Sep 16 07:47:00.800620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.799926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz4wf\" (UniqueName: \"kubernetes.io/projected/8999f450-ddca-4885-af15-c203abdf47d7-kube-api-access-sz4wf\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.800620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.799983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-workload-socket\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.800620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800015    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-envoy\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.800620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-workload-certs\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.800620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-podinfo\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.800620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8999f450-ddca-4885-af15-c203abdf47d7-istiod-ca-cert\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.800620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800158    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-istio-envoy\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.800620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-token\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.800620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-credential-socket\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.800620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8999f450-ddca-4885-af15-c203abdf47d7-istio-token\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.800620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800284    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-workload-certs\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.800620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdzt4\" (UniqueName: \"kubernetes.io/projected/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-kube-api-access-hdzt4\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.800620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-data\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.800620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istiod-ca-cert\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.801369 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8999f450-ddca-4885-af15-c203abdf47d7-istio-podinfo\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.801369 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-credential-socket\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.801369 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800472    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-workload-socket\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.801369 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-istio-data\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.801369 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.800950    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-istio-data\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.801369 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.801305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-credential-socket\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.801677 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.801564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-workload-socket\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.801898 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.801875    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-workload-certs\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.802354 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.802329    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8999f450-ddca-4885-af15-c203abdf47d7-istiod-ca-cert\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.806930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.806905    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8999f450-ddca-4885-af15-c203abdf47d7-istio-podinfo\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.807203 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.807156    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-istio-envoy\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.810836 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.810436    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz4wf\" (UniqueName: \"kubernetes.io/projected/8999f450-ddca-4885-af15-c203abdf47d7-kube-api-access-sz4wf\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.810836 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.810799    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8999f450-ddca-4885-af15-c203abdf47d7-istio-token\") pod \"e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") " pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:00.854783 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.854744    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:00.901967 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.901646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-workload-socket\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.901967 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.901725    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-envoy\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.901967 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.901768    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-podinfo\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.901967 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.901860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-token\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.901967 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.901895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-workload-certs\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.901967 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.901926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdzt4\" (UniqueName: \"kubernetes.io/projected/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-kube-api-access-hdzt4\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.901967 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.901954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-data\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.902350 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.901992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istiod-ca-cert\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.902350 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.902023    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-credential-socket\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.902350 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.902215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-workload-socket\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.902350 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.902306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-credential-socket\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.903598 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.902452    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-workload-certs\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.903598 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.902864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-data\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.903598 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.903473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istiod-ca-cert\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.906548 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.906402    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-podinfo\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.907419 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.907045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-envoy\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.917894 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.912146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-token\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.917894 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.912990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdzt4\" (UniqueName: \"kubernetes.io/projected/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-kube-api-access-hdzt4\") pod \"e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") " pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:00.933418 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:00.933231    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:01.036947 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:01.036917    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:01.129851 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:01.129804    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"]
Sep 16 07:47:01.131596 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:47:01.131575    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60a8a8f_b2a4_40e7_88fb_ffee17f9a01b.slice/crio-bbf1bc672cee355529a4f2f6483f4e4dbd3c7dafe5c34da02532b5beb5b1f67a WatchSource:0}: Error finding container bbf1bc672cee355529a4f2f6483f4e4dbd3c7dafe5c34da02532b5beb5b1f67a: Status 404 returned error can't find the container with id bbf1bc672cee355529a4f2f6483f4e4dbd3c7dafe5c34da02532b5beb5b1f67a
Sep 16 07:47:01.133921 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:01.133889    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:01.133978 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:01.133959    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:01.134010 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:01.133988    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:01.211867 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:01.211838    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"]
Sep 16 07:47:01.213063 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:47:01.213032    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8999f450_ddca_4885_af15_c203abdf47d7.slice/crio-5bea0f225ee72bb262df5085f2537843603f781cd3883241dba946707ef1964a WatchSource:0}: Error finding container 5bea0f225ee72bb262df5085f2537843603f781cd3883241dba946707ef1964a: Status 404 returned error can't find the container with id 5bea0f225ee72bb262df5085f2537843603f781cd3883241dba946707ef1964a
Sep 16 07:47:01.216455 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:01.216415    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:01.216575 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:01.216510    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:01.216575 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:01.216556    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:01.315909 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:01.315875    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"]
Sep 16 07:47:01.317787 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:47:01.317761    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b309cf1_ec9d_4d4d_808b_ef1f4d54a236.slice/crio-379d808fb4142b2e9adea9a26d5ac2d0830219405c9f9ca49e52df1034bb2f3f WatchSource:0}: Error finding container 379d808fb4142b2e9adea9a26d5ac2d0830219405c9f9ca49e52df1034bb2f3f: Status 404 returned error can't find the container with id 379d808fb4142b2e9adea9a26d5ac2d0830219405c9f9ca49e52df1034bb2f3f
Sep 16 07:47:01.320963 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:01.320931    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:01.321064 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:01.321010    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:01.321064 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:01.321051    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:02.098491 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:02.098450    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr" event={"ID":"8999f450-ddca-4885-af15-c203abdf47d7","Type":"ContainerStarted","Data":"cef60180dd544efa6550d3548da28f32d6165b814f2253ea03045276f6821750"}
Sep 16 07:47:02.098959 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:02.098544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr" event={"ID":"8999f450-ddca-4885-af15-c203abdf47d7","Type":"ContainerStarted","Data":"5bea0f225ee72bb262df5085f2537843603f781cd3883241dba946707ef1964a"}
Sep 16 07:47:02.100141 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:02.100119    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c" event={"ID":"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b","Type":"ContainerStarted","Data":"33f02d1e40305976363c2776efc7b306cc12123359f6debb47a5b1cbc10d7bf1"}
Sep 16 07:47:02.100239 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:02.100162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c" event={"ID":"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b","Type":"ContainerStarted","Data":"bbf1bc672cee355529a4f2f6483f4e4dbd3c7dafe5c34da02532b5beb5b1f67a"}
Sep 16 07:47:02.101549 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:02.101528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp" event={"ID":"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236","Type":"ContainerStarted","Data":"68262c4d2c267115c5fb3dd0a0ee9d5e34e638124e1cec17f0a17c3da697dd7c"}
Sep 16 07:47:02.101638 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:02.101566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp" event={"ID":"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236","Type":"ContainerStarted","Data":"379d808fb4142b2e9adea9a26d5ac2d0830219405c9f9ca49e52df1034bb2f3f"}
Sep 16 07:47:02.122141 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:02.122101    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr" podStartSLOduration=2.122087771 podStartE2EDuration="2.122087771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:02.120494542 +0000 UTC m=+783.047106697" watchObservedRunningTime="2026-09-16 07:47:02.122087771 +0000 UTC m=+783.048699923"
Sep 16 07:47:02.140491 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:02.140448    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c" podStartSLOduration=2.140434878 podStartE2EDuration="2.140434878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:02.138465854 +0000 UTC m=+783.065078007" watchObservedRunningTime="2026-09-16 07:47:02.140434878 +0000 UTC m=+783.067047031"
Sep 16 07:47:02.158064 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:02.158019    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp" podStartSLOduration=2.158004776 podStartE2EDuration="2.158004776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:02.156312194 +0000 UTC m=+783.082924349" watchObservedRunningTime="2026-09-16 07:47:02.158004776 +0000 UTC m=+783.084616929"
Sep 16 07:47:02.855565 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:02.855536    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:02.860545 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:02.860521    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:02.933288 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:02.933266    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:02.937780 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:02.937756    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:03.037681 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:03.037656    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:03.042282 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:03.042261    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:03.106377 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:03.106304    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:03.106377 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:03.106347    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:03.106377 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:03.106368    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:03.107240 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:03.107220    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:03.107374 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:03.107360    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:03.107589 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:03.107575    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:08.156156 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.156124    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"]
Sep 16 07:47:08.158448 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.158430    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"
Sep 16 07:47:08.161233 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.161208    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5dd30f-dockercfg-nwwbw\""
Sep 16 07:47:08.161471 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.161449    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5dd30f\""
Sep 16 07:47:08.170487 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.170465    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"]
Sep 16 07:47:08.188457 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.188431    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"]
Sep 16 07:47:08.191107 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.191089    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"
Sep 16 07:47:08.206538 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.206514    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"]
Sep 16 07:47:08.276471 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.276420    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5l96\" (UniqueName: \"kubernetes.io/projected/ac0b949d-481f-4dba-8da8-08adfabc40ee-kube-api-access-r5l96\") pod \"payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8\" (UID: \"ac0b949d-481f-4dba-8da8-08adfabc40ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"
Sep 16 07:47:08.276653 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.276495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e9831a-5cea-44bf-8e19-0670680f33c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv\" (UID: \"d5e9831a-5cea-44bf-8e19-0670680f33c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"
Sep 16 07:47:08.276653 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.276551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0b949d-481f-4dba-8da8-08adfabc40ee-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8\" (UID: \"ac0b949d-481f-4dba-8da8-08adfabc40ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"
Sep 16 07:47:08.276653 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.276638    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m69qd\" (UniqueName: \"kubernetes.io/projected/d5e9831a-5cea-44bf-8e19-0670680f33c8-kube-api-access-m69qd\") pod \"payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv\" (UID: \"d5e9831a-5cea-44bf-8e19-0670680f33c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"
Sep 16 07:47:08.377605 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.377571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69qd\" (UniqueName: \"kubernetes.io/projected/d5e9831a-5cea-44bf-8e19-0670680f33c8-kube-api-access-m69qd\") pod \"payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv\" (UID: \"d5e9831a-5cea-44bf-8e19-0670680f33c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"
Sep 16 07:47:08.377755 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.377623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5l96\" (UniqueName: \"kubernetes.io/projected/ac0b949d-481f-4dba-8da8-08adfabc40ee-kube-api-access-r5l96\") pod \"payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8\" (UID: \"ac0b949d-481f-4dba-8da8-08adfabc40ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"
Sep 16 07:47:08.377755 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.377658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e9831a-5cea-44bf-8e19-0670680f33c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv\" (UID: \"d5e9831a-5cea-44bf-8e19-0670680f33c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"
Sep 16 07:47:08.377755 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.377688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0b949d-481f-4dba-8da8-08adfabc40ee-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8\" (UID: \"ac0b949d-481f-4dba-8da8-08adfabc40ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"
Sep 16 07:47:08.378362 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.378342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e9831a-5cea-44bf-8e19-0670680f33c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv\" (UID: \"d5e9831a-5cea-44bf-8e19-0670680f33c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"
Sep 16 07:47:08.378416 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.378382    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0b949d-481f-4dba-8da8-08adfabc40ee-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8\" (UID: \"ac0b949d-481f-4dba-8da8-08adfabc40ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"
Sep 16 07:47:08.385617 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.385593    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5l96\" (UniqueName: \"kubernetes.io/projected/ac0b949d-481f-4dba-8da8-08adfabc40ee-kube-api-access-r5l96\") pod \"payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8\" (UID: \"ac0b949d-481f-4dba-8da8-08adfabc40ee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"
Sep 16 07:47:08.385725 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.385656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m69qd\" (UniqueName: \"kubernetes.io/projected/d5e9831a-5cea-44bf-8e19-0670680f33c8-kube-api-access-m69qd\") pod \"payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv\" (UID: \"d5e9831a-5cea-44bf-8e19-0670680f33c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"
Sep 16 07:47:08.470654 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.470580    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"
Sep 16 07:47:08.500653 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.500621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"
Sep 16 07:47:08.749169 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.749141    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"]
Sep 16 07:47:08.750187 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:47:08.750163    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e9831a_5cea_44bf_8e19_0670680f33c8.slice/crio-c0a5403785d143a3f2ce903905d9dbf7533500691822fab4946ddfc6b55c8bdc WatchSource:0}: Error finding container c0a5403785d143a3f2ce903905d9dbf7533500691822fab4946ddfc6b55c8bdc: Status 404 returned error can't find the container with id c0a5403785d143a3f2ce903905d9dbf7533500691822fab4946ddfc6b55c8bdc
Sep 16 07:47:08.801887 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:08.801862    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"]
Sep 16 07:47:08.804291 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:47:08.804267    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0b949d_481f_4dba_8da8_08adfabc40ee.slice/crio-fea7a320edb593f09b1f3eaac3d692cb9fbaeb1983c38945fe69a03163d08599 WatchSource:0}: Error finding container fea7a320edb593f09b1f3eaac3d692cb9fbaeb1983c38945fe69a03163d08599: Status 404 returned error can't find the container with id fea7a320edb593f09b1f3eaac3d692cb9fbaeb1983c38945fe69a03163d08599
Sep 16 07:47:09.143374 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.142987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8" event={"ID":"ac0b949d-481f-4dba-8da8-08adfabc40ee","Type":"ContainerStarted","Data":"4960207c8f6c77c4f83f7138daaf89e3add9700a4f3df837cac0e3f5f028fc94"}
Sep 16 07:47:09.143374 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.143059    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8" event={"ID":"ac0b949d-481f-4dba-8da8-08adfabc40ee","Type":"ContainerStarted","Data":"fea7a320edb593f09b1f3eaac3d692cb9fbaeb1983c38945fe69a03163d08599"}
Sep 16 07:47:09.143374 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.143112    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"
Sep 16 07:47:09.145212 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.145117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv" event={"ID":"d5e9831a-5cea-44bf-8e19-0670680f33c8","Type":"ContainerStarted","Data":"d7c7cffc5a8aa5ae83ee5d67923ee7047c2b4f5dd6a410eff33eef5e12f91940"}
Sep 16 07:47:09.145212 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.145210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv" event={"ID":"d5e9831a-5cea-44bf-8e19-0670680f33c8","Type":"ContainerStarted","Data":"c0a5403785d143a3f2ce903905d9dbf7533500691822fab4946ddfc6b55c8bdc"}
Sep 16 07:47:09.145406 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.145239    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"
Sep 16 07:47:09.167082 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.167036    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8" podStartSLOduration=1.1670205359999999 podStartE2EDuration="1.167020536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:09.165702034 +0000 UTC m=+790.092314184" watchObservedRunningTime="2026-09-16 07:47:09.167020536 +0000 UTC m=+790.093632741"
Sep 16 07:47:09.427659 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.427587    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv" podStartSLOduration=1.427564481 podStartE2EDuration="1.427564481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:09.19680701 +0000 UTC m=+790.123419164" watchObservedRunningTime="2026-09-16 07:47:09.427564481 +0000 UTC m=+790.354176637"
Sep 16 07:47:09.430386 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.430357    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"]
Sep 16 07:47:09.436332 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.436311    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.439620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.439596    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0de2c4d3-openshift-default-dockercfg-nz6fn\""
Sep 16 07:47:09.449015 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.447557    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"]
Sep 16 07:47:09.492635 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.492584    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6592eff5-28fe-4022-aac1-62815734741c-istio-podinfo\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.492635 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.492634    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-istio-envoy\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.492872 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.492671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6592eff5-28fe-4022-aac1-62815734741c-istio-token\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.492872 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.492711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-istio-data\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.492872 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.492771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-workload-certs\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.492872 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.492848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-workload-socket\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.493046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.492883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwhck\" (UniqueName: \"kubernetes.io/projected/6592eff5-28fe-4022-aac1-62815734741c-kube-api-access-dwhck\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.493046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.492918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-credential-socket\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.493046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.492970    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6592eff5-28fe-4022-aac1-62815734741c-istiod-ca-cert\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.593858 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.593802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-istio-data\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.594026 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.593882    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-workload-certs\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.594026 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.593948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-workload-socket\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.594026 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.593980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwhck\" (UniqueName: \"kubernetes.io/projected/6592eff5-28fe-4022-aac1-62815734741c-kube-api-access-dwhck\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.594026 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.594012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-credential-socket\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.594240 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.594154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6592eff5-28fe-4022-aac1-62815734741c-istiod-ca-cert\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.594297 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.594242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6592eff5-28fe-4022-aac1-62815734741c-istio-podinfo\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.594297 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.594279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-istio-envoy\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.594396 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.594312    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6592eff5-28fe-4022-aac1-62815734741c-istio-token\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.594396 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.594319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-istio-data\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.594533 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.594509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-credential-socket\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.594670 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.594643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-workload-socket\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.594755 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.594563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-workload-certs\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.595145 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.595122    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6592eff5-28fe-4022-aac1-62815734741c-istiod-ca-cert\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.596627 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.596603    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-istio-envoy\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.596896 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.596874    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6592eff5-28fe-4022-aac1-62815734741c-istio-podinfo\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.601427 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.601398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6592eff5-28fe-4022-aac1-62815734741c-istio-token\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.601684 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.601665    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwhck\" (UniqueName: \"kubernetes.io/projected/6592eff5-28fe-4022-aac1-62815734741c-kube-api-access-dwhck\") pod \"e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") " pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:09.751948 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:09.751866    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:10.055097 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:10.055070    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"]
Sep 16 07:47:10.055345 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:47:10.055317    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6592eff5_28fe_4022_aac1_62815734741c.slice/crio-b12f03e416ac866dabb491979577e4b9b8b7c6b2aed064fa20671ed8437c678f WatchSource:0}: Error finding container b12f03e416ac866dabb491979577e4b9b8b7c6b2aed064fa20671ed8437c678f: Status 404 returned error can't find the container with id b12f03e416ac866dabb491979577e4b9b8b7c6b2aed064fa20671ed8437c678f
Sep 16 07:47:10.057781 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:10.057752    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:10.057881 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:10.057811    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:10.057881 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:10.057860    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:10.162144 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:10.162104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn" event={"ID":"6592eff5-28fe-4022-aac1-62815734741c","Type":"ContainerStarted","Data":"e142f032ef7ebdfb4d7c08ad256b0f56c650e7b93202821df21409a6e0328789"}
Sep 16 07:47:10.162317 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:10.162196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn" event={"ID":"6592eff5-28fe-4022-aac1-62815734741c","Type":"ContainerStarted","Data":"b12f03e416ac866dabb491979577e4b9b8b7c6b2aed064fa20671ed8437c678f"}
Sep 16 07:47:10.201733 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:10.201662    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn" podStartSLOduration=1.201644273 podStartE2EDuration="1.201644273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:10.200546167 +0000 UTC m=+791.127158321" watchObservedRunningTime="2026-09-16 07:47:10.201644273 +0000 UTC m=+791.128256428"
Sep 16 07:47:10.752007 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:10.751973    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:11.757476 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:11.757442    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:12.174550 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:12.174511    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:12.176046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:12.176008    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:15.888268 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:15.888230    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"]
Sep 16 07:47:15.890698 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:15.890682    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"
Sep 16 07:47:15.893118 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:15.893081    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0de2c4d3\""
Sep 16 07:47:15.893240 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:15.893214    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0de2c4d3-dockercfg-m8zfd\""
Sep 16 07:47:15.902031 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:15.902011    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"]
Sep 16 07:47:15.986657 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:15.982019    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"]
Sep 16 07:47:15.986657 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:15.982955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvvhp\" (UniqueName: \"kubernetes.io/projected/7dc21c86-e2f7-4738-98b1-761da71b537b-kube-api-access-wvvhp\") pod \"payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59\" (UID: \"7dc21c86-e2f7-4738-98b1-761da71b537b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"
Sep 16 07:47:15.986657 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:15.983035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc21c86-e2f7-4738-98b1-761da71b537b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59\" (UID: \"7dc21c86-e2f7-4738-98b1-761da71b537b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"
Sep 16 07:47:15.986657 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:15.985559    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"
Sep 16 07:47:15.998377 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:15.998210    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"]
Sep 16 07:47:16.084516 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:16.084481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvvhp\" (UniqueName: \"kubernetes.io/projected/7dc21c86-e2f7-4738-98b1-761da71b537b-kube-api-access-wvvhp\") pod \"payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59\" (UID: \"7dc21c86-e2f7-4738-98b1-761da71b537b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"
Sep 16 07:47:16.084709 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:16.084555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc21c86-e2f7-4738-98b1-761da71b537b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59\" (UID: \"7dc21c86-e2f7-4738-98b1-761da71b537b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"
Sep 16 07:47:16.084709 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:16.084600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rczng\" (UniqueName: \"kubernetes.io/projected/94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6-kube-api-access-rczng\") pod \"payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w\" (UID: \"94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6\") " pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"
Sep 16 07:47:16.084709 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:16.084663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6-plugins-config-volume\") pod \"payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w\" (UID: \"94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6\") " pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"
Sep 16 07:47:16.085575 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:16.085551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc21c86-e2f7-4738-98b1-761da71b537b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59\" (UID: \"7dc21c86-e2f7-4738-98b1-761da71b537b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"
Sep 16 07:47:16.093865 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:16.093195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvvhp\" (UniqueName: \"kubernetes.io/projected/7dc21c86-e2f7-4738-98b1-761da71b537b-kube-api-access-wvvhp\") pod \"payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59\" (UID: \"7dc21c86-e2f7-4738-98b1-761da71b537b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"
Sep 16 07:47:16.185361 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:16.185322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczng\" (UniqueName: \"kubernetes.io/projected/94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6-kube-api-access-rczng\") pod \"payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w\" (UID: \"94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6\") " pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"
Sep 16 07:47:16.185532 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:16.185385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6-plugins-config-volume\") pod \"payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w\" (UID: \"94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6\") " pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"
Sep 16 07:47:16.186125 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:16.186105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6-plugins-config-volume\") pod \"payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w\" (UID: \"94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6\") " pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"
Sep 16 07:47:16.195367 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:16.195341    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rczng\" (UniqueName: \"kubernetes.io/projected/94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6-kube-api-access-rczng\") pod \"payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w\" (UID: \"94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6\") " pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"
Sep 16 07:47:16.202175 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:16.202153    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"
Sep 16 07:47:16.302856 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:16.302065    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"
Sep 16 07:47:16.534528 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:47:16.534475    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc21c86_e2f7_4738_98b1_761da71b537b.slice/crio-240649d8d261bcaabdf1132786cf39315dcee7ec43d5bdb1db976ec329ee0ce9 WatchSource:0}: Error finding container 240649d8d261bcaabdf1132786cf39315dcee7ec43d5bdb1db976ec329ee0ce9: Status 404 returned error can't find the container with id 240649d8d261bcaabdf1132786cf39315dcee7ec43d5bdb1db976ec329ee0ce9
Sep 16 07:47:16.539032 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:16.537801    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"]
Sep 16 07:47:16.673463 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:16.673437    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"]
Sep 16 07:47:16.675611 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:47:16.675563    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94eb7bc3_d7dd_4d51_815e_9bac6d9a15c6.slice/crio-cb9688c589fcf8d27bb063e8652b2f150efc4d4e6412d5d91457c0ecf12c5f86 WatchSource:0}: Error finding container cb9688c589fcf8d27bb063e8652b2f150efc4d4e6412d5d91457c0ecf12c5f86: Status 404 returned error can't find the container with id cb9688c589fcf8d27bb063e8652b2f150efc4d4e6412d5d91457c0ecf12c5f86
Sep 16 07:47:17.203500 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:17.203451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59" event={"ID":"7dc21c86-e2f7-4738-98b1-761da71b537b","Type":"ContainerStarted","Data":"e81fe3fc53f398c835c560a813915c4d03979e69ded55892448abaa13bfc9cee"}
Sep 16 07:47:17.203987 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:17.203511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59" event={"ID":"7dc21c86-e2f7-4738-98b1-761da71b537b","Type":"ContainerStarted","Data":"240649d8d261bcaabdf1132786cf39315dcee7ec43d5bdb1db976ec329ee0ce9"}
Sep 16 07:47:17.203987 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:17.203920    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"
Sep 16 07:47:17.205451 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:17.205427    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w" event={"ID":"94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6","Type":"ContainerStarted","Data":"69ebb493f2b1fc488feacafb96e521f38b282aab0f39125420927c6157797e0e"}
Sep 16 07:47:17.205556 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:17.205476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w" event={"ID":"94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6","Type":"ContainerStarted","Data":"cb9688c589fcf8d27bb063e8652b2f150efc4d4e6412d5d91457c0ecf12c5f86"}
Sep 16 07:47:17.205598 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:17.205554    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"
Sep 16 07:47:17.223511 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:17.223444    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59" podStartSLOduration=2.223424082 podStartE2EDuration="2.223424082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:17.220353416 +0000 UTC m=+798.146965565" watchObservedRunningTime="2026-09-16 07:47:17.223424082 +0000 UTC m=+798.150036237"
Sep 16 07:47:17.241560 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:17.241400    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w" podStartSLOduration=2.241383555 podStartE2EDuration="2.241383555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:17.238028696 +0000 UTC m=+798.164640849" watchObservedRunningTime="2026-09-16 07:47:17.241383555 +0000 UTC m=+798.167995709"
Sep 16 07:47:19.149033 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:19.148990    2581 patch_prober.go:28] interesting pod/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 07:47:19.149614 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:19.149076    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp" podUID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 07:47:19.163648 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:19.163610    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"]
Sep 16 07:47:19.166939 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:19.166785    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"
Sep 16 07:47:19.170314 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:19.170284    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-b60434\""
Sep 16 07:47:19.170439 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:19.170401    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-b60434-dockercfg-srcx6\""
Sep 16 07:47:19.181927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:19.181900    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"]
Sep 16 07:47:19.347708 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:19.347668    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2jrv\" (UniqueName: \"kubernetes.io/projected/53f89282-047a-4a97-a603-fff05b8eda4b-kube-api-access-q2jrv\") pod \"payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g\" (UID: \"53f89282-047a-4a97-a603-fff05b8eda4b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"
Sep 16 07:47:19.347900 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:19.347776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f89282-047a-4a97-a603-fff05b8eda4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g\" (UID: \"53f89282-047a-4a97-a603-fff05b8eda4b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"
Sep 16 07:47:19.448797 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:19.448750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f89282-047a-4a97-a603-fff05b8eda4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g\" (UID: \"53f89282-047a-4a97-a603-fff05b8eda4b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"
Sep 16 07:47:19.449022 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:19.448909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2jrv\" (UniqueName: \"kubernetes.io/projected/53f89282-047a-4a97-a603-fff05b8eda4b-kube-api-access-q2jrv\") pod \"payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g\" (UID: \"53f89282-047a-4a97-a603-fff05b8eda4b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"
Sep 16 07:47:19.449469 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:19.449445    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f89282-047a-4a97-a603-fff05b8eda4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g\" (UID: \"53f89282-047a-4a97-a603-fff05b8eda4b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"
Sep 16 07:47:19.464205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:19.464178    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2jrv\" (UniqueName: \"kubernetes.io/projected/53f89282-047a-4a97-a603-fff05b8eda4b-kube-api-access-q2jrv\") pod \"payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g\" (UID: \"53f89282-047a-4a97-a603-fff05b8eda4b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"
Sep 16 07:47:19.481960 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:19.481936    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"
Sep 16 07:47:19.777291 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:19.777267    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"]
Sep 16 07:47:19.779669 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:47:19.779623    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f89282_047a_4a97_a603_fff05b8eda4b.slice/crio-ceb778a837e9c73806f8e6eb2eaae32523a2338abc8283dd20781c6198a23dfe WatchSource:0}: Error finding container ceb778a837e9c73806f8e6eb2eaae32523a2338abc8283dd20781c6198a23dfe: Status 404 returned error can't find the container with id ceb778a837e9c73806f8e6eb2eaae32523a2338abc8283dd20781c6198a23dfe
Sep 16 07:47:20.164690 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:20.164652    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"
Sep 16 07:47:20.165172 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:20.164872    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"
Sep 16 07:47:20.235873 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:20.235835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g" event={"ID":"53f89282-047a-4a97-a603-fff05b8eda4b","Type":"ContainerStarted","Data":"95cfff12441bf2908e8c8144a9a0cbe5545fee34a7a24f756bc7cb1579d7f508"}
Sep 16 07:47:20.236133 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:20.236112    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g" event={"ID":"53f89282-047a-4a97-a603-fff05b8eda4b","Type":"ContainerStarted","Data":"ceb778a837e9c73806f8e6eb2eaae32523a2338abc8283dd20781c6198a23dfe"}
Sep 16 07:47:20.236990 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:20.236969    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"
Sep 16 07:47:20.259674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:20.259610    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g" podStartSLOduration=1.259586391 podStartE2EDuration="1.259586391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:20.258407915 +0000 UTC m=+801.185020074" watchObservedRunningTime="2026-09-16 07:47:20.259586391 +0000 UTC m=+801.186198539"
Sep 16 07:47:23.259919 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:23.259883    2581 generic.go:356] "Generic (PLEG): container finished" podID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerID="68262c4d2c267115c5fb3dd0a0ee9d5e34e638124e1cec17f0a17c3da697dd7c" exitCode=0
Sep 16 07:47:23.261461 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:23.260014    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp" event={"ID":"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236","Type":"ContainerDied","Data":"68262c4d2c267115c5fb3dd0a0ee9d5e34e638124e1cec17f0a17c3da697dd7c"}
Sep 16 07:47:23.261461 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:23.261065    2581 scope.go:117] "RemoveContainer" containerID="68262c4d2c267115c5fb3dd0a0ee9d5e34e638124e1cec17f0a17c3da697dd7c"
Sep 16 07:47:23.262523 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:23.262481    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:23.262627 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:23.262571    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:23.262627 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:23.262621    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:24.270645 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:24.270589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp" event={"ID":"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236","Type":"ContainerStarted","Data":"1f8b616d3dea9802949515cdd5a1baec743317cf2dbfac3368ce54c9f25a2fce"}
Sep 16 07:47:25.038024 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:25.037583    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:25.038024 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:25.037709    2581 patch_prober.go:28] interesting pod/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 07:47:25.038024 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:25.037753    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp" podUID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 07:47:26.038301 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:26.037854    2581 patch_prober.go:28] interesting pod/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 07:47:26.038301 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:26.037920    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp" podUID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 07:47:26.286716 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:26.286679    2581 generic.go:356] "Generic (PLEG): container finished" podID="a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" containerID="33f02d1e40305976363c2776efc7b306cc12123359f6debb47a5b1cbc10d7bf1" exitCode=0
Sep 16 07:47:26.286912 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:26.286755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c" event={"ID":"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b","Type":"ContainerDied","Data":"33f02d1e40305976363c2776efc7b306cc12123359f6debb47a5b1cbc10d7bf1"}
Sep 16 07:47:26.287759 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:26.287235    2581 scope.go:117] "RemoveContainer" containerID="33f02d1e40305976363c2776efc7b306cc12123359f6debb47a5b1cbc10d7bf1"
Sep 16 07:47:26.288570 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:26.288539    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:26.288665 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:26.288614    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:26.288665 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:26.288647    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:27.041842 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:27.040415    2581 patch_prober.go:28] interesting pod/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 07:47:27.041842 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:27.040475    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp" podUID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 07:47:27.294203 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:27.294033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c" event={"ID":"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b","Type":"ContainerStarted","Data":"d7ceffd5d8b9890e3b08d56b575cbdebc3887f29144a32b18248a1914a550957"}
Sep 16 07:47:27.856098 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:27.855608    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:27.856098 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:27.855740    2581 patch_prober.go:28] interesting pod/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 16 07:47:27.856098 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:27.855784    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c" podUID="a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 16 07:47:28.037650 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:28.037602    2581 patch_prober.go:28] interesting pod/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 07:47:28.037870 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:28.037678    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp" podUID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 07:47:28.217020 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:28.216980    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"
Sep 16 07:47:28.217515 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:28.216987    2581 patch_prober.go:28] interesting pod/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 07:47:28.217515 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:28.217084    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn" podUID="6592eff5-28fe-4022-aac1-62815734741c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 07:47:28.856337 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:28.855863    2581 patch_prober.go:28] interesting pod/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 16 07:47:28.856337 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:28.855926    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c" podUID="a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 16 07:47:29.038083 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:29.038037    2581 patch_prober.go:28] interesting pod/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 07:47:29.038270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:29.038108    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp" podUID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 07:47:29.224403 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:29.224075    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"
Sep 16 07:47:29.855325 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:29.855283    2581 patch_prober.go:28] interesting pod/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 16 07:47:29.855524 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:29.855370    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c" podUID="a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 16 07:47:30.038037 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:30.037560    2581 patch_prober.go:28] interesting pod/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 16 07:47:30.038037 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:30.037628    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp" podUID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 16 07:47:30.855715 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:30.855675    2581 patch_prober.go:28] interesting pod/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 16 07:47:30.856225 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:30.855733    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c" podUID="a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 16 07:47:31.043970 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:31.043942    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:31.044387 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:31.044363    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:31.045125 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:31.045105    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:47:31.324240 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:31.324196    2581 generic.go:356] "Generic (PLEG): container finished" podID="6592eff5-28fe-4022-aac1-62815734741c" containerID="e142f032ef7ebdfb4d7c08ad256b0f56c650e7b93202821df21409a6e0328789" exitCode=0
Sep 16 07:47:31.325083 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:31.325053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn" event={"ID":"6592eff5-28fe-4022-aac1-62815734741c","Type":"ContainerDied","Data":"e142f032ef7ebdfb4d7c08ad256b0f56c650e7b93202821df21409a6e0328789"}
Sep 16 07:47:31.325496 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:31.325473    2581 scope.go:117] "RemoveContainer" containerID="e142f032ef7ebdfb4d7c08ad256b0f56c650e7b93202821df21409a6e0328789"
Sep 16 07:47:31.329199 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:31.329166    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:31.329864 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:31.329839    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:31.330032 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:31.330013    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:31.855230 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:31.855125    2581 patch_prober.go:28] interesting pod/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 16 07:47:31.855230 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:31.855192    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c" podUID="a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 16 07:47:32.254061 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:32.254016    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"
Sep 16 07:47:32.350844 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:32.347095    2581 generic.go:356] "Generic (PLEG): container finished" podID="8999f450-ddca-4885-af15-c203abdf47d7" containerID="cef60180dd544efa6550d3548da28f32d6165b814f2253ea03045276f6821750" exitCode=0
Sep 16 07:47:32.350844 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:32.347221    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr" event={"ID":"8999f450-ddca-4885-af15-c203abdf47d7","Type":"ContainerDied","Data":"cef60180dd544efa6550d3548da28f32d6165b814f2253ea03045276f6821750"}
Sep 16 07:47:32.350844 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:32.347752    2581 scope.go:117] "RemoveContainer" containerID="cef60180dd544efa6550d3548da28f32d6165b814f2253ea03045276f6821750"
Sep 16 07:47:32.354853 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:32.353958    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:32.354853 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:32.354055    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:32.354853 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:32.354102    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:32.378850 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:32.375310    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn" event={"ID":"6592eff5-28fe-4022-aac1-62815734741c","Type":"ContainerStarted","Data":"2f5a7a150607fa5dbd8fbb3cfbdd052502767f1812d04b8adcda8e968b10208a"}
Sep 16 07:47:32.752872 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:32.752420    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:32.752872 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:32.752546    2581 patch_prober.go:28] interesting pod/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 16 07:47:32.752872 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:32.752590    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn" podUID="6592eff5-28fe-4022-aac1-62815734741c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 16 07:47:32.932464 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:32.932427    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:32.932909 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:32.932848    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:32.936511 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:32.936482    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 07:47:33.107221 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:33.107076    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:33.385187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:33.385050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr" event={"ID":"8999f450-ddca-4885-af15-c203abdf47d7","Type":"ContainerStarted","Data":"b08d1465f1e10724dcb4c93dc687f936aa2d0d48e48254f12185ad98a670f0bb"}
Sep 16 07:47:33.752479 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:33.752445    2581 patch_prober.go:28] interesting pod/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 16 07:47:33.752768 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:33.752741    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn" podUID="6592eff5-28fe-4022-aac1-62815734741c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 16 07:47:33.934920 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:33.934877    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:33.935118 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:33.935054    2581 patch_prober.go:28] interesting pod/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 16 07:47:33.935118 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:33.935103    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr" podUID="8999f450-ddca-4885-af15-c203abdf47d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 16 07:47:34.753120 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:34.752640    2581 patch_prober.go:28] interesting pod/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 16 07:47:34.753120 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:34.752719    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn" podUID="6592eff5-28fe-4022-aac1-62815734741c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 16 07:47:34.934285 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:34.934240    2581 patch_prober.go:28] interesting pod/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 16 07:47:34.934479 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:34.934309    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr" podUID="8999f450-ddca-4885-af15-c203abdf47d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 16 07:47:35.753802 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:35.753763    2581 patch_prober.go:28] interesting pod/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 16 07:47:35.754303 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:35.753849    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn" podUID="6592eff5-28fe-4022-aac1-62815734741c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 16 07:47:35.933985 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:35.933945    2581 patch_prober.go:28] interesting pod/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 16 07:47:35.934170 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:35.934010    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr" podUID="8999f450-ddca-4885-af15-c203abdf47d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 16 07:47:36.752454 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:36.752403    2581 patch_prober.go:28] interesting pod/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 16 07:47:36.752643 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:36.752479    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn" podUID="6592eff5-28fe-4022-aac1-62815734741c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 16 07:47:36.933980 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:36.933945    2581 patch_prober.go:28] interesting pod/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 16 07:47:36.934549 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:36.934505    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr" podUID="8999f450-ddca-4885-af15-c203abdf47d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 16 07:47:37.756657 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:37.756624    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:37.756957 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:37.756933    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:37.757916 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:37.757897    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:37.934208 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:37.934165    2581 patch_prober.go:28] interesting pod/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 16 07:47:37.934680 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:37.934233    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr" podUID="8999f450-ddca-4885-af15-c203abdf47d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 16 07:47:38.937691 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:38.937661    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:38.938086 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:38.937925    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:38.938486 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:38.938468    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 07:47:42.875016 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:42.874987    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"]
Sep 16 07:47:42.875443 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:42.875192    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w" podUID="94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6" containerName="payload-processing" containerID="cri-o://69ebb493f2b1fc488feacafb96e521f38b282aab0f39125420927c6157797e0e" gracePeriod=30
Sep 16 07:47:42.881163 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:42.881138    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"]
Sep 16 07:47:42.881354 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:42.881324    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59" podUID="7dc21c86-e2f7-4738-98b1-761da71b537b" containerName="payload-pre-processing" containerID="cri-o://e81fe3fc53f398c835c560a813915c4d03979e69ded55892448abaa13bfc9cee" gracePeriod=30
Sep 16 07:47:43.170851 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.158955    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"]
Sep 16 07:47:43.170851 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.159182    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn" podUID="6592eff5-28fe-4022-aac1-62815734741c" containerName="istio-proxy" containerID="cri-o://2f5a7a150607fa5dbd8fbb3cfbdd052502767f1812d04b8adcda8e968b10208a" gracePeriod=30
Sep 16 07:47:43.217809 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.217777    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"
Sep 16 07:47:43.221013 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.220993    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"
Sep 16 07:47:43.295457 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.295249    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc21c86-e2f7-4738-98b1-761da71b537b-plugins-config-volume\") pod \"7dc21c86-e2f7-4738-98b1-761da71b537b\" (UID: \"7dc21c86-e2f7-4738-98b1-761da71b537b\") "
Sep 16 07:47:43.298842 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.297525    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc21c86-e2f7-4738-98b1-761da71b537b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7dc21c86-e2f7-4738-98b1-761da71b537b" (UID: "7dc21c86-e2f7-4738-98b1-761da71b537b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:47:43.298842 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.297719    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvvhp\" (UniqueName: \"kubernetes.io/projected/7dc21c86-e2f7-4738-98b1-761da71b537b-kube-api-access-wvvhp\") pod \"7dc21c86-e2f7-4738-98b1-761da71b537b\" (UID: \"7dc21c86-e2f7-4738-98b1-761da71b537b\") "
Sep 16 07:47:43.298842 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.297776    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6-plugins-config-volume\") pod \"94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6\" (UID: \"94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6\") "
Sep 16 07:47:43.298842 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.297834    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rczng\" (UniqueName: \"kubernetes.io/projected/94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6-kube-api-access-rczng\") pod \"94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6\" (UID: \"94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6\") "
Sep 16 07:47:43.298842 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.298267    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc21c86-e2f7-4738-98b1-761da71b537b-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:47:43.302710 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.301013    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6" (UID: "94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:47:43.306697 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.306666    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6-kube-api-access-rczng" (OuterVolumeSpecName: "kube-api-access-rczng") pod "94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6" (UID: "94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6"). InnerVolumeSpecName "kube-api-access-rczng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:47:43.306862 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.306839    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc21c86-e2f7-4738-98b1-761da71b537b-kube-api-access-wvvhp" (OuterVolumeSpecName: "kube-api-access-wvvhp") pod "7dc21c86-e2f7-4738-98b1-761da71b537b" (UID: "7dc21c86-e2f7-4738-98b1-761da71b537b"). InnerVolumeSpecName "kube-api-access-wvvhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:47:43.399070 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.399040    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvvhp\" (UniqueName: \"kubernetes.io/projected/7dc21c86-e2f7-4738-98b1-761da71b537b-kube-api-access-wvvhp\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:47:43.399070 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.399064    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:47:43.399070 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.399074    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rczng\" (UniqueName: \"kubernetes.io/projected/94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6-kube-api-access-rczng\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:47:43.444902 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.444809    2581 generic.go:356] "Generic (PLEG): container finished" podID="94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6" containerID="69ebb493f2b1fc488feacafb96e521f38b282aab0f39125420927c6157797e0e" exitCode=0
Sep 16 07:47:43.444902 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.444843    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w" event={"ID":"94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6","Type":"ContainerDied","Data":"69ebb493f2b1fc488feacafb96e521f38b282aab0f39125420927c6157797e0e"}
Sep 16 07:47:43.444902 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.444868    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"
Sep 16 07:47:43.445139 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.444907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w" event={"ID":"94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6","Type":"ContainerDied","Data":"cb9688c589fcf8d27bb063e8652b2f150efc4d4e6412d5d91457c0ecf12c5f86"}
Sep 16 07:47:43.445139 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.444941    2581 scope.go:117] "RemoveContainer" containerID="69ebb493f2b1fc488feacafb96e521f38b282aab0f39125420927c6157797e0e"
Sep 16 07:47:43.446263 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.446237    2581 generic.go:356] "Generic (PLEG): container finished" podID="7dc21c86-e2f7-4738-98b1-761da71b537b" containerID="e81fe3fc53f398c835c560a813915c4d03979e69ded55892448abaa13bfc9cee" exitCode=0
Sep 16 07:47:43.446386 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.446286    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"
Sep 16 07:47:43.446386 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.446288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59" event={"ID":"7dc21c86-e2f7-4738-98b1-761da71b537b","Type":"ContainerDied","Data":"e81fe3fc53f398c835c560a813915c4d03979e69ded55892448abaa13bfc9cee"}
Sep 16 07:47:43.446386 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.446326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59" event={"ID":"7dc21c86-e2f7-4738-98b1-761da71b537b","Type":"ContainerDied","Data":"240649d8d261bcaabdf1132786cf39315dcee7ec43d5bdb1db976ec329ee0ce9"}
Sep 16 07:47:43.454987 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.454968    2581 scope.go:117] "RemoveContainer" containerID="69ebb493f2b1fc488feacafb96e521f38b282aab0f39125420927c6157797e0e"
Sep 16 07:47:43.455263 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:47:43.455247    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ebb493f2b1fc488feacafb96e521f38b282aab0f39125420927c6157797e0e\": container with ID starting with 69ebb493f2b1fc488feacafb96e521f38b282aab0f39125420927c6157797e0e not found: ID does not exist" containerID="69ebb493f2b1fc488feacafb96e521f38b282aab0f39125420927c6157797e0e"
Sep 16 07:47:43.455324 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.455268    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ebb493f2b1fc488feacafb96e521f38b282aab0f39125420927c6157797e0e"} err="failed to get container status \"69ebb493f2b1fc488feacafb96e521f38b282aab0f39125420927c6157797e0e\": rpc error: code = NotFound desc = could not find container \"69ebb493f2b1fc488feacafb96e521f38b282aab0f39125420927c6157797e0e\": container with ID starting with 69ebb493f2b1fc488feacafb96e521f38b282aab0f39125420927c6157797e0e not found: ID does not exist"
Sep 16 07:47:43.455324 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.455283    2581 scope.go:117] "RemoveContainer" containerID="e81fe3fc53f398c835c560a813915c4d03979e69ded55892448abaa13bfc9cee"
Sep 16 07:47:43.462867 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.462848    2581 scope.go:117] "RemoveContainer" containerID="e81fe3fc53f398c835c560a813915c4d03979e69ded55892448abaa13bfc9cee"
Sep 16 07:47:43.463141 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:47:43.463122    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e81fe3fc53f398c835c560a813915c4d03979e69ded55892448abaa13bfc9cee\": container with ID starting with e81fe3fc53f398c835c560a813915c4d03979e69ded55892448abaa13bfc9cee not found: ID does not exist" containerID="e81fe3fc53f398c835c560a813915c4d03979e69ded55892448abaa13bfc9cee"
Sep 16 07:47:43.463199 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.463147    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e81fe3fc53f398c835c560a813915c4d03979e69ded55892448abaa13bfc9cee"} err="failed to get container status \"e81fe3fc53f398c835c560a813915c4d03979e69ded55892448abaa13bfc9cee\": rpc error: code = NotFound desc = could not find container \"e81fe3fc53f398c835c560a813915c4d03979e69ded55892448abaa13bfc9cee\": container with ID starting with e81fe3fc53f398c835c560a813915c4d03979e69ded55892448abaa13bfc9cee not found: ID does not exist"
Sep 16 07:47:43.467271 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.467250    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"]
Sep 16 07:47:43.471185 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.471167    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0de2c4d3-f57b9cc97-5z68w"]
Sep 16 07:47:43.480606 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.480580    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"]
Sep 16 07:47:43.483804 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.483783    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0de2c4d3-76576995d-wdv59"]
Sep 16 07:47:43.684565 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.684534    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dc21c86-e2f7-4738-98b1-761da71b537b" path="/var/lib/kubelet/pods/7dc21c86-e2f7-4738-98b1-761da71b537b/volumes"
Sep 16 07:47:43.684945 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:43.684932    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6" path="/var/lib/kubelet/pods/94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6/volumes"
Sep 16 07:47:44.123986 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.123949    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"]
Sep 16 07:47:44.124571 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.124555    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6" containerName="payload-processing"
Sep 16 07:47:44.124613 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.124575    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6" containerName="payload-processing"
Sep 16 07:47:44.124613 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.124597    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dc21c86-e2f7-4738-98b1-761da71b537b" containerName="payload-pre-processing"
Sep 16 07:47:44.124613 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.124605    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dc21c86-e2f7-4738-98b1-761da71b537b" containerName="payload-pre-processing"
Sep 16 07:47:44.124715 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.124704    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="94eb7bc3-d7dd-4d51-815e-9bac6d9a15c6" containerName="payload-processing"
Sep 16 07:47:44.124754 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.124723    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dc21c86-e2f7-4738-98b1-761da71b537b" containerName="payload-pre-processing"
Sep 16 07:47:44.128537 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.128514    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.130955 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.130934    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e5fb973f-gw-openshift-default-dockercfg-bgpz7\""
Sep 16 07:47:44.139960 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.139940    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"]
Sep 16 07:47:44.211806 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.211773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bbs\" (UniqueName: \"kubernetes.io/projected/41c1d84b-657f-46b5-a767-97a20bbe614f-kube-api-access-v9bbs\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.211973 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.211867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-envoy\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.211973 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.211926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c1d84b-657f-46b5-a767-97a20bbe614f-istiod-ca-cert\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.212093 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.212033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-data\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.212147 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.212109    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-workload-certs\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.212199 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.212148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-credential-socket\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.212199 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.212173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-workload-socket\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.212289 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.212227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-token\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.212289 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.212260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-podinfo\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.313735 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.313696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-token\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.313943 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.313747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-podinfo\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.313943 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.313789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bbs\" (UniqueName: \"kubernetes.io/projected/41c1d84b-657f-46b5-a767-97a20bbe614f-kube-api-access-v9bbs\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.313943 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.313838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-envoy\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.313943 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.313876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c1d84b-657f-46b5-a767-97a20bbe614f-istiod-ca-cert\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.314157 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.313936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-data\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.314157 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.314009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-workload-certs\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.314157 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.314047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-credential-socket\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.314157 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.314074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-workload-socket\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.314606 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.314580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-workload-socket\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.314736 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.314705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-data\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.314834 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.314793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c1d84b-657f-46b5-a767-97a20bbe614f-istiod-ca-cert\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.314989 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.314969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-workload-certs\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.315129 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.315109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-credential-socket\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.316855 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.316835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-podinfo\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.316951 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.316934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-envoy\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.322515 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.322489    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-token\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.322652 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.322635    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bbs\" (UniqueName: \"kubernetes.io/projected/41c1d84b-657f-46b5-a767-97a20bbe614f-kube-api-access-v9bbs\") pod \"e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") " pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.440628 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.440594    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:44.718349 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.718325    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"]
Sep 16 07:47:44.719551 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:47:44.719527    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c1d84b_657f_46b5_a767_97a20bbe614f.slice/crio-08054d3f92ac1bd7ca5acec646ad9917429598f3f2575b72192a9effb6c0feec WatchSource:0}: Error finding container 08054d3f92ac1bd7ca5acec646ad9917429598f3f2575b72192a9effb6c0feec: Status 404 returned error can't find the container with id 08054d3f92ac1bd7ca5acec646ad9917429598f3f2575b72192a9effb6c0feec
Sep 16 07:47:44.721719 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.721688    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:44.721833 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.721748    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:44.721833 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:44.721776    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:47:45.461345 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:45.461308    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d" event={"ID":"41c1d84b-657f-46b5-a767-97a20bbe614f","Type":"ContainerStarted","Data":"2c4c1d97bb9458dffd83d489bbdf8d7f1eaa8710f4c3abd65a824f889f2adee7"}
Sep 16 07:47:45.461803 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:45.461396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d" event={"ID":"41c1d84b-657f-46b5-a767-97a20bbe614f","Type":"ContainerStarted","Data":"08054d3f92ac1bd7ca5acec646ad9917429598f3f2575b72192a9effb6c0feec"}
Sep 16 07:47:45.485052 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:45.485005    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d" podStartSLOduration=1.484990879 podStartE2EDuration="1.484990879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:45.482378078 +0000 UTC m=+826.408990232" watchObservedRunningTime="2026-09-16 07:47:45.484990879 +0000 UTC m=+826.411603032"
Sep 16 07:47:46.441760 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:46.441729    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:46.447432 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:46.447409    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:46.466563 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:46.466533    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:46.467236 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:46.467200    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:47:48.482288 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.482261    2581 generic.go:356] "Generic (PLEG): container finished" podID="6592eff5-28fe-4022-aac1-62815734741c" containerID="2f5a7a150607fa5dbd8fbb3cfbdd052502767f1812d04b8adcda8e968b10208a" exitCode=0
Sep 16 07:47:48.482628 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.482335    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn" event={"ID":"6592eff5-28fe-4022-aac1-62815734741c","Type":"ContainerDied","Data":"2f5a7a150607fa5dbd8fbb3cfbdd052502767f1812d04b8adcda8e968b10208a"}
Sep 16 07:47:48.482628 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.482416    2581 scope.go:117] "RemoveContainer" containerID="e142f032ef7ebdfb4d7c08ad256b0f56c650e7b93202821df21409a6e0328789"
Sep 16 07:47:48.528482 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.528464    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:48.658724 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.658701    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-credential-socket\") pod \"6592eff5-28fe-4022-aac1-62815734741c\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") "
Sep 16 07:47:48.658883 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.658736    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6592eff5-28fe-4022-aac1-62815734741c-istio-podinfo\") pod \"6592eff5-28fe-4022-aac1-62815734741c\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") "
Sep 16 07:47:48.658883 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.658763    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-workload-certs\") pod \"6592eff5-28fe-4022-aac1-62815734741c\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") "
Sep 16 07:47:48.658883 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.658793    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6592eff5-28fe-4022-aac1-62815734741c-istiod-ca-cert\") pod \"6592eff5-28fe-4022-aac1-62815734741c\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") "
Sep 16 07:47:48.658883 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.658840    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-istio-data\") pod \"6592eff5-28fe-4022-aac1-62815734741c\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") "
Sep 16 07:47:48.659104 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.658894    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-istio-envoy\") pod \"6592eff5-28fe-4022-aac1-62815734741c\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") "
Sep 16 07:47:48.659104 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.658933    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwhck\" (UniqueName: \"kubernetes.io/projected/6592eff5-28fe-4022-aac1-62815734741c-kube-api-access-dwhck\") pod \"6592eff5-28fe-4022-aac1-62815734741c\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") "
Sep 16 07:47:48.659104 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.658960    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6592eff5-28fe-4022-aac1-62815734741c-istio-token\") pod \"6592eff5-28fe-4022-aac1-62815734741c\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") "
Sep 16 07:47:48.659104 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.658998    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-workload-socket\") pod \"6592eff5-28fe-4022-aac1-62815734741c\" (UID: \"6592eff5-28fe-4022-aac1-62815734741c\") "
Sep 16 07:47:48.659104 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.658995    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6592eff5-28fe-4022-aac1-62815734741c" (UID: "6592eff5-28fe-4022-aac1-62815734741c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:47:48.659349 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.659196    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6592eff5-28fe-4022-aac1-62815734741c" (UID: "6592eff5-28fe-4022-aac1-62815734741c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:47:48.659349 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.659236    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-istio-data" (OuterVolumeSpecName: "istio-data") pod "6592eff5-28fe-4022-aac1-62815734741c" (UID: "6592eff5-28fe-4022-aac1-62815734741c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:47:48.659349 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.659343    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:47:48.659505 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.659361    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:47:48.659505 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.659376    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:47:48.659505 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.659404    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6592eff5-28fe-4022-aac1-62815734741c" (UID: "6592eff5-28fe-4022-aac1-62815734741c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:47:48.659505 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.659445    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6592eff5-28fe-4022-aac1-62815734741c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6592eff5-28fe-4022-aac1-62815734741c" (UID: "6592eff5-28fe-4022-aac1-62815734741c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:47:48.661124 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.661102    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6592eff5-28fe-4022-aac1-62815734741c-istio-token" (OuterVolumeSpecName: "istio-token") pod "6592eff5-28fe-4022-aac1-62815734741c" (UID: "6592eff5-28fe-4022-aac1-62815734741c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:47:48.661240 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.661162    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6592eff5-28fe-4022-aac1-62815734741c-kube-api-access-dwhck" (OuterVolumeSpecName: "kube-api-access-dwhck") pod "6592eff5-28fe-4022-aac1-62815734741c" (UID: "6592eff5-28fe-4022-aac1-62815734741c"). InnerVolumeSpecName "kube-api-access-dwhck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:47:48.661466 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.661451    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6592eff5-28fe-4022-aac1-62815734741c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6592eff5-28fe-4022-aac1-62815734741c" (UID: "6592eff5-28fe-4022-aac1-62815734741c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 07:47:48.661466 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.661455    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6592eff5-28fe-4022-aac1-62815734741c" (UID: "6592eff5-28fe-4022-aac1-62815734741c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:47:48.760312 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.760287    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6592eff5-28fe-4022-aac1-62815734741c-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:47:48.760312 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.760311    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6592eff5-28fe-4022-aac1-62815734741c-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:47:48.760469 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.760320    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:47:48.760469 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.760328    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwhck\" (UniqueName: \"kubernetes.io/projected/6592eff5-28fe-4022-aac1-62815734741c-kube-api-access-dwhck\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:47:48.760469 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.760337    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6592eff5-28fe-4022-aac1-62815734741c-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:47:48.760469 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:48.760345    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592eff5-28fe-4022-aac1-62815734741c-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:47:49.491239 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:49.491209    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"
Sep 16 07:47:49.491674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:49.491255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn" event={"ID":"6592eff5-28fe-4022-aac1-62815734741c","Type":"ContainerDied","Data":"b12f03e416ac866dabb491979577e4b9b8b7c6b2aed064fa20671ed8437c678f"}
Sep 16 07:47:49.491674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:49.491311    2581 scope.go:117] "RemoveContainer" containerID="2f5a7a150607fa5dbd8fbb3cfbdd052502767f1812d04b8adcda8e968b10208a"
Sep 16 07:47:49.517426 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:49.517402    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"]
Sep 16 07:47:49.521377 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:49.521356    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0de2c4d3-openshift-default-d9598b5c4-6wffn"]
Sep 16 07:47:49.684447 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:49.684405    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6592eff5-28fe-4022-aac1-62815734741c" path="/var/lib/kubelet/pods/6592eff5-28fe-4022-aac1-62815734741c/volumes"
Sep 16 07:47:50.939412 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.939379    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"]
Sep 16 07:47:50.939862 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.939777    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6592eff5-28fe-4022-aac1-62815734741c" containerName="istio-proxy"
Sep 16 07:47:50.939862 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.939788    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6592eff5-28fe-4022-aac1-62815734741c" containerName="istio-proxy"
Sep 16 07:47:50.939862 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.939795    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6592eff5-28fe-4022-aac1-62815734741c" containerName="istio-proxy"
Sep 16 07:47:50.939862 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.939800    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6592eff5-28fe-4022-aac1-62815734741c" containerName="istio-proxy"
Sep 16 07:47:50.939994 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.939874    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6592eff5-28fe-4022-aac1-62815734741c" containerName="istio-proxy"
Sep 16 07:47:50.939994 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.939883    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6592eff5-28fe-4022-aac1-62815734741c" containerName="istio-proxy"
Sep 16 07:47:50.943466 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.943438    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"
Sep 16 07:47:50.946067 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.946047    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e5fb973f\""
Sep 16 07:47:50.946200 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.946072    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e5fb973f-dockercfg-dn8b4\""
Sep 16 07:47:50.957372 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.957352    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"]
Sep 16 07:47:50.959612 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.959591    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"]
Sep 16 07:47:50.959701 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.959685    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"
Sep 16 07:47:50.972377 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.972354    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"]
Sep 16 07:47:50.982951 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.982922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhzxh\" (UniqueName: \"kubernetes.io/projected/2025360b-5f2d-4a5c-9448-0d57d5af6c05-kube-api-access-jhzxh\") pod \"payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv\" (UID: \"2025360b-5f2d-4a5c-9448-0d57d5af6c05\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"
Sep 16 07:47:50.983117 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:50.983100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2025360b-5f2d-4a5c-9448-0d57d5af6c05-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv\" (UID: \"2025360b-5f2d-4a5c-9448-0d57d5af6c05\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"
Sep 16 07:47:51.083909 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:51.083875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2025360b-5f2d-4a5c-9448-0d57d5af6c05-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv\" (UID: \"2025360b-5f2d-4a5c-9448-0d57d5af6c05\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"
Sep 16 07:47:51.084105 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:51.084004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57a5b27-a825-406b-9bb2-30ac477c18a5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv\" (UID: \"a57a5b27-a825-406b-9bb2-30ac477c18a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"
Sep 16 07:47:51.084105 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:51.084053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhzxh\" (UniqueName: \"kubernetes.io/projected/2025360b-5f2d-4a5c-9448-0d57d5af6c05-kube-api-access-jhzxh\") pod \"payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv\" (UID: \"2025360b-5f2d-4a5c-9448-0d57d5af6c05\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"
Sep 16 07:47:51.084185 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:51.084144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dzdn\" (UniqueName: \"kubernetes.io/projected/a57a5b27-a825-406b-9bb2-30ac477c18a5-kube-api-access-2dzdn\") pod \"payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv\" (UID: \"a57a5b27-a825-406b-9bb2-30ac477c18a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"
Sep 16 07:47:51.084547 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:51.084529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2025360b-5f2d-4a5c-9448-0d57d5af6c05-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv\" (UID: \"2025360b-5f2d-4a5c-9448-0d57d5af6c05\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"
Sep 16 07:47:51.091420 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:51.091391    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhzxh\" (UniqueName: \"kubernetes.io/projected/2025360b-5f2d-4a5c-9448-0d57d5af6c05-kube-api-access-jhzxh\") pod \"payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv\" (UID: \"2025360b-5f2d-4a5c-9448-0d57d5af6c05\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"
Sep 16 07:47:51.184972 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:51.184945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dzdn\" (UniqueName: \"kubernetes.io/projected/a57a5b27-a825-406b-9bb2-30ac477c18a5-kube-api-access-2dzdn\") pod \"payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv\" (UID: \"a57a5b27-a825-406b-9bb2-30ac477c18a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"
Sep 16 07:47:51.185095 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:51.185030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57a5b27-a825-406b-9bb2-30ac477c18a5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv\" (UID: \"a57a5b27-a825-406b-9bb2-30ac477c18a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"
Sep 16 07:47:51.185707 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:51.185685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57a5b27-a825-406b-9bb2-30ac477c18a5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv\" (UID: \"a57a5b27-a825-406b-9bb2-30ac477c18a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"
Sep 16 07:47:51.193217 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:51.193170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dzdn\" (UniqueName: \"kubernetes.io/projected/a57a5b27-a825-406b-9bb2-30ac477c18a5-kube-api-access-2dzdn\") pod \"payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv\" (UID: \"a57a5b27-a825-406b-9bb2-30ac477c18a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"
Sep 16 07:47:51.254274 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:51.254249    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"
Sep 16 07:47:51.269593 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:51.269562    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"
Sep 16 07:47:51.524389 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:51.524366    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"]
Sep 16 07:47:51.526923 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:47:51.526899    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2025360b_5f2d_4a5c_9448_0d57d5af6c05.slice/crio-b8249783ee44fabdde103426857aa429faab12de137effcd21787e2bc3adf27c WatchSource:0}: Error finding container b8249783ee44fabdde103426857aa429faab12de137effcd21787e2bc3adf27c: Status 404 returned error can't find the container with id b8249783ee44fabdde103426857aa429faab12de137effcd21787e2bc3adf27c
Sep 16 07:47:51.559940 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:51.559916    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"]
Sep 16 07:47:51.562052 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:47:51.562032    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57a5b27_a825_406b_9bb2_30ac477c18a5.slice/crio-2c24ea2ebe236866ffcdd938e4629c40b478aea2d657aeefc78c1a42f8a1aae9 WatchSource:0}: Error finding container 2c24ea2ebe236866ffcdd938e4629c40b478aea2d657aeefc78c1a42f8a1aae9: Status 404 returned error can't find the container with id 2c24ea2ebe236866ffcdd938e4629c40b478aea2d657aeefc78c1a42f8a1aae9
Sep 16 07:47:52.508262 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:52.508224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv" event={"ID":"2025360b-5f2d-4a5c-9448-0d57d5af6c05","Type":"ContainerStarted","Data":"3c54db6428715685a0fde999ae52a0c1631e176c8bcc4e2539dc4dfe5c9f7736"}
Sep 16 07:47:52.508666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:52.508281    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv" event={"ID":"2025360b-5f2d-4a5c-9448-0d57d5af6c05","Type":"ContainerStarted","Data":"b8249783ee44fabdde103426857aa429faab12de137effcd21787e2bc3adf27c"}
Sep 16 07:47:52.508666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:52.508326    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"
Sep 16 07:47:52.509884 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:52.509859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv" event={"ID":"a57a5b27-a825-406b-9bb2-30ac477c18a5","Type":"ContainerStarted","Data":"5920063a818510bd5eef4ad762c19854ec66b84b652a5f575ff252da4101a563"}
Sep 16 07:47:52.510021 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:52.509902    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv" event={"ID":"a57a5b27-a825-406b-9bb2-30ac477c18a5","Type":"ContainerStarted","Data":"2c24ea2ebe236866ffcdd938e4629c40b478aea2d657aeefc78c1a42f8a1aae9"}
Sep 16 07:47:52.510021 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:52.509946    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"
Sep 16 07:47:52.532221 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:52.532167    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv" podStartSLOduration=2.5321485040000002 podStartE2EDuration="2.532148504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:52.528308361 +0000 UTC m=+833.454920517" watchObservedRunningTime="2026-09-16 07:47:52.532148504 +0000 UTC m=+833.458760659"
Sep 16 07:47:52.547099 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:47:52.547058    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv" podStartSLOduration=2.547045066 podStartE2EDuration="2.547045066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:52.546987737 +0000 UTC m=+833.473599891" watchObservedRunningTime="2026-09-16 07:47:52.547045066 +0000 UTC m=+833.473657495"
Sep 16 07:48:03.519586 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:03.519542    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"
Sep 16 07:48:03.520959 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:03.520940    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"
Sep 16 07:48:10.602984 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:10.602953    2581 generic.go:356] "Generic (PLEG): container finished" podID="41c1d84b-657f-46b5-a767-97a20bbe614f" containerID="2c4c1d97bb9458dffd83d489bbdf8d7f1eaa8710f4c3abd65a824f889f2adee7" exitCode=0
Sep 16 07:48:10.603371 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:10.603023    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d" event={"ID":"41c1d84b-657f-46b5-a767-97a20bbe614f","Type":"ContainerDied","Data":"2c4c1d97bb9458dffd83d489bbdf8d7f1eaa8710f4c3abd65a824f889f2adee7"}
Sep 16 07:48:10.603412 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:10.603403    2581 scope.go:117] "RemoveContainer" containerID="2c4c1d97bb9458dffd83d489bbdf8d7f1eaa8710f4c3abd65a824f889f2adee7"
Sep 16 07:48:10.604583 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:10.604556    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:48:10.604677 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:10.604624    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:48:10.604677 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:10.604658    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:48:11.611073 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:11.611030    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d" event={"ID":"41c1d84b-657f-46b5-a767-97a20bbe614f","Type":"ContainerStarted","Data":"c73d0dcde01c9939895c8386fbdf6336d3e3e6039bf7066f338af128e54c5182"}
Sep 16 07:48:12.441021 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:12.440974    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:48:12.442135 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:12.442111    2581 patch_prober.go:28] interesting pod/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 16 07:48:12.442198 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:12.442164    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d" podUID="41c1d84b-657f-46b5-a767-97a20bbe614f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 16 07:48:13.441272 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:13.441233    2581 patch_prober.go:28] interesting pod/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 16 07:48:13.441670 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:13.441308    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d" podUID="41c1d84b-657f-46b5-a767-97a20bbe614f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 16 07:48:14.441234 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:14.441202    2581 patch_prober.go:28] interesting pod/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 16 07:48:14.441398 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:14.441251    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d" podUID="41c1d84b-657f-46b5-a767-97a20bbe614f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 16 07:48:15.441610 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:15.441576    2581 patch_prober.go:28] interesting pod/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 16 07:48:15.441995 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:15.441626    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d" podUID="41c1d84b-657f-46b5-a767-97a20bbe614f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 16 07:48:16.441167 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:16.441136    2581 patch_prober.go:28] interesting pod/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 16 07:48:16.441332 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:16.441193    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d" podUID="41c1d84b-657f-46b5-a767-97a20bbe614f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 16 07:48:16.466717 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:16.466689    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:48:16.942235 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:16.942203    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg"]
Sep 16 07:48:16.944795 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:16.944778    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg"
Sep 16 07:48:16.947087 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:16.947064    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 07:48:16.947202 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:16.947069    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\""
Sep 16 07:48:16.952743 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:16.952718    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg"]
Sep 16 07:48:17.075526 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:17.075495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6115c556-efd0-40c3-8868-224f02669064-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg\" (UID: \"6115c556-efd0-40c3-8868-224f02669064\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg"
Sep 16 07:48:17.176132 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:17.176095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6115c556-efd0-40c3-8868-224f02669064-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg\" (UID: \"6115c556-efd0-40c3-8868-224f02669064\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg"
Sep 16 07:48:17.176662 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:17.176644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6115c556-efd0-40c3-8868-224f02669064-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg\" (UID: \"6115c556-efd0-40c3-8868-224f02669064\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg"
Sep 16 07:48:17.262312 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:17.262217    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg"
Sep 16 07:48:17.487694 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:17.487659    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:48:17.489507 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:17.489477    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:48:17.533999 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:17.533951    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg"]
Sep 16 07:48:17.535323 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:48:17.535279    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6115c556_efd0_40c3_8868_224f02669064.slice/crio-868e83d824592a88519ba8f3f92d66d0d66bff6f8f3363cbe19c8dd77f87fa88 WatchSource:0}: Error finding container 868e83d824592a88519ba8f3f92d66d0d66bff6f8f3363cbe19c8dd77f87fa88: Status 404 returned error can't find the container with id 868e83d824592a88519ba8f3f92d66d0d66bff6f8f3363cbe19c8dd77f87fa88
Sep 16 07:48:17.652194 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:17.652157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg" event={"ID":"6115c556-efd0-40c3-8868-224f02669064","Type":"ContainerStarted","Data":"868e83d824592a88519ba8f3f92d66d0d66bff6f8f3363cbe19c8dd77f87fa88"}
Sep 16 07:48:20.672177 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:20.672145    2581 generic.go:356] "Generic (PLEG): container finished" podID="6115c556-efd0-40c3-8868-224f02669064" containerID="c9710708bcb1bc311b5d6c1b3a8507ee150f0a4350419c0dd7e5c07a9e114777" exitCode=0
Sep 16 07:48:20.672546 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:20.672228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg" event={"ID":"6115c556-efd0-40c3-8868-224f02669064","Type":"ContainerDied","Data":"c9710708bcb1bc311b5d6c1b3a8507ee150f0a4350419c0dd7e5c07a9e114777"}
Sep 16 07:48:21.805257 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:21.805232    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg"
Sep 16 07:48:21.921764 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:21.921731    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6115c556-efd0-40c3-8868-224f02669064-maas-api-service-ca\") pod \"6115c556-efd0-40c3-8868-224f02669064\" (UID: \"6115c556-efd0-40c3-8868-224f02669064\") "
Sep 16 07:48:21.922093 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:21.922068    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6115c556-efd0-40c3-8868-224f02669064-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6115c556-efd0-40c3-8868-224f02669064" (UID: "6115c556-efd0-40c3-8868-224f02669064"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:48:22.023021 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:22.022953    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6115c556-efd0-40c3-8868-224f02669064-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:48:22.682376 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:22.682348    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg"
Sep 16 07:48:22.682531 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:22.682380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg" event={"ID":"6115c556-efd0-40c3-8868-224f02669064","Type":"ContainerDied","Data":"868e83d824592a88519ba8f3f92d66d0d66bff6f8f3363cbe19c8dd77f87fa88"}
Sep 16 07:48:22.682531 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:22.682415    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="868e83d824592a88519ba8f3f92d66d0d66bff6f8f3363cbe19c8dd77f87fa88"
Sep 16 07:48:28.570111 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.570076    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"]
Sep 16 07:48:28.570572 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.570354    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv" podUID="a57a5b27-a825-406b-9bb2-30ac477c18a5" containerName="payload-processing" containerID="cri-o://5920063a818510bd5eef4ad762c19854ec66b84b652a5f575ff252da4101a563" gracePeriod=30
Sep 16 07:48:28.579671 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.579624    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"]
Sep 16 07:48:28.579887 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.579858    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv" podUID="2025360b-5f2d-4a5c-9448-0d57d5af6c05" containerName="payload-pre-processing" containerID="cri-o://3c54db6428715685a0fde999ae52a0c1631e176c8bcc4e2539dc4dfe5c9f7736" gracePeriod=30
Sep 16 07:48:28.716020 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.715992    2581 generic.go:356] "Generic (PLEG): container finished" podID="2025360b-5f2d-4a5c-9448-0d57d5af6c05" containerID="3c54db6428715685a0fde999ae52a0c1631e176c8bcc4e2539dc4dfe5c9f7736" exitCode=0
Sep 16 07:48:28.716234 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.716072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv" event={"ID":"2025360b-5f2d-4a5c-9448-0d57d5af6c05","Type":"ContainerDied","Data":"3c54db6428715685a0fde999ae52a0c1631e176c8bcc4e2539dc4dfe5c9f7736"}
Sep 16 07:48:28.717959 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.717768    2581 generic.go:356] "Generic (PLEG): container finished" podID="a57a5b27-a825-406b-9bb2-30ac477c18a5" containerID="5920063a818510bd5eef4ad762c19854ec66b84b652a5f575ff252da4101a563" exitCode=0
Sep 16 07:48:28.717959 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.717804    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv" event={"ID":"a57a5b27-a825-406b-9bb2-30ac477c18a5","Type":"ContainerDied","Data":"5920063a818510bd5eef4ad762c19854ec66b84b652a5f575ff252da4101a563"}
Sep 16 07:48:28.760471 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.760441    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg"]
Sep 16 07:48:28.764911 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.764887    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e5fb973f-47htg"]
Sep 16 07:48:28.856863 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.856841    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"
Sep 16 07:48:28.860338 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.860318    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"
Sep 16 07:48:28.947705 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.947672    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"]
Sep 16 07:48:28.947893 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.947873    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d" podUID="41c1d84b-657f-46b5-a767-97a20bbe614f" containerName="istio-proxy" containerID="cri-o://c73d0dcde01c9939895c8386fbdf6336d3e3e6039bf7066f338af128e54c5182" gracePeriod=30
Sep 16 07:48:28.990499 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.990468    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2025360b-5f2d-4a5c-9448-0d57d5af6c05-plugins-config-volume\") pod \"2025360b-5f2d-4a5c-9448-0d57d5af6c05\" (UID: \"2025360b-5f2d-4a5c-9448-0d57d5af6c05\") "
Sep 16 07:48:28.990705 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.990653    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dzdn\" (UniqueName: \"kubernetes.io/projected/a57a5b27-a825-406b-9bb2-30ac477c18a5-kube-api-access-2dzdn\") pod \"a57a5b27-a825-406b-9bb2-30ac477c18a5\" (UID: \"a57a5b27-a825-406b-9bb2-30ac477c18a5\") "
Sep 16 07:48:28.990705 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.990690    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57a5b27-a825-406b-9bb2-30ac477c18a5-plugins-config-volume\") pod \"a57a5b27-a825-406b-9bb2-30ac477c18a5\" (UID: \"a57a5b27-a825-406b-9bb2-30ac477c18a5\") "
Sep 16 07:48:28.990870 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.990733    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhzxh\" (UniqueName: \"kubernetes.io/projected/2025360b-5f2d-4a5c-9448-0d57d5af6c05-kube-api-access-jhzxh\") pod \"2025360b-5f2d-4a5c-9448-0d57d5af6c05\" (UID: \"2025360b-5f2d-4a5c-9448-0d57d5af6c05\") "
Sep 16 07:48:28.990992 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.990961    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2025360b-5f2d-4a5c-9448-0d57d5af6c05-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2025360b-5f2d-4a5c-9448-0d57d5af6c05" (UID: "2025360b-5f2d-4a5c-9448-0d57d5af6c05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:48:28.991135 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.991095    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2025360b-5f2d-4a5c-9448-0d57d5af6c05-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:48:28.991238 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.991124    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57a5b27-a825-406b-9bb2-30ac477c18a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a57a5b27-a825-406b-9bb2-30ac477c18a5" (UID: "a57a5b27-a825-406b-9bb2-30ac477c18a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:48:28.992953 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.992910    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2025360b-5f2d-4a5c-9448-0d57d5af6c05-kube-api-access-jhzxh" (OuterVolumeSpecName: "kube-api-access-jhzxh") pod "2025360b-5f2d-4a5c-9448-0d57d5af6c05" (UID: "2025360b-5f2d-4a5c-9448-0d57d5af6c05"). InnerVolumeSpecName "kube-api-access-jhzxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:48:28.993190 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:28.993167    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57a5b27-a825-406b-9bb2-30ac477c18a5-kube-api-access-2dzdn" (OuterVolumeSpecName: "kube-api-access-2dzdn") pod "a57a5b27-a825-406b-9bb2-30ac477c18a5" (UID: "a57a5b27-a825-406b-9bb2-30ac477c18a5"). InnerVolumeSpecName "kube-api-access-2dzdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:48:29.092608 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.092532    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57a5b27-a825-406b-9bb2-30ac477c18a5-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:48:29.092608 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.092556    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhzxh\" (UniqueName: \"kubernetes.io/projected/2025360b-5f2d-4a5c-9448-0d57d5af6c05-kube-api-access-jhzxh\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:48:29.092608 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.092566    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dzdn\" (UniqueName: \"kubernetes.io/projected/a57a5b27-a825-406b-9bb2-30ac477c18a5-kube-api-access-2dzdn\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:48:29.670091 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.670058    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"]
Sep 16 07:48:29.670548 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.670506    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a57a5b27-a825-406b-9bb2-30ac477c18a5" containerName="payload-processing"
Sep 16 07:48:29.670548 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.670524    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a57a5b27-a825-406b-9bb2-30ac477c18a5" containerName="payload-processing"
Sep 16 07:48:29.670665 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.670552    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2025360b-5f2d-4a5c-9448-0d57d5af6c05" containerName="payload-pre-processing"
Sep 16 07:48:29.670665 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.670592    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2025360b-5f2d-4a5c-9448-0d57d5af6c05" containerName="payload-pre-processing"
Sep 16 07:48:29.670665 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.670627    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6115c556-efd0-40c3-8868-224f02669064" containerName="revoke-keys"
Sep 16 07:48:29.670665 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.670636    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6115c556-efd0-40c3-8868-224f02669064" containerName="revoke-keys"
Sep 16 07:48:29.670937 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.670767    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2025360b-5f2d-4a5c-9448-0d57d5af6c05" containerName="payload-pre-processing"
Sep 16 07:48:29.670937 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.670785    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a57a5b27-a825-406b-9bb2-30ac477c18a5" containerName="payload-processing"
Sep 16 07:48:29.670937 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.670797    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6115c556-efd0-40c3-8868-224f02669064" containerName="revoke-keys"
Sep 16 07:48:29.673312 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.673285    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.675628 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.675608    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4d9f69e4-openshift-default-dockercfg-xxq8x\""
Sep 16 07:48:29.686117 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.686089    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6115c556-efd0-40c3-8868-224f02669064" path="/var/lib/kubelet/pods/6115c556-efd0-40c3-8868-224f02669064/volumes"
Sep 16 07:48:29.686722 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.686701    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"]
Sep 16 07:48:29.724847 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.724811    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"
Sep 16 07:48:29.724847 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.724829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv" event={"ID":"2025360b-5f2d-4a5c-9448-0d57d5af6c05","Type":"ContainerDied","Data":"b8249783ee44fabdde103426857aa429faab12de137effcd21787e2bc3adf27c"}
Sep 16 07:48:29.725028 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.724897    2581 scope.go:117] "RemoveContainer" containerID="3c54db6428715685a0fde999ae52a0c1631e176c8bcc4e2539dc4dfe5c9f7736"
Sep 16 07:48:29.726390 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.726356    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"
Sep 16 07:48:29.726390 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.726367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv" event={"ID":"a57a5b27-a825-406b-9bb2-30ac477c18a5","Type":"ContainerDied","Data":"2c24ea2ebe236866ffcdd938e4629c40b478aea2d657aeefc78c1a42f8a1aae9"}
Sep 16 07:48:29.734946 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.734925    2581 scope.go:117] "RemoveContainer" containerID="5920063a818510bd5eef4ad762c19854ec66b84b652a5f575ff252da4101a563"
Sep 16 07:48:29.744745 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.744723    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"]
Sep 16 07:48:29.747072 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.747047    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e5fb973f-6687f8ccbb-f4jhv"]
Sep 16 07:48:29.757953 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.757930    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"]
Sep 16 07:48:29.769014 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.767757    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e5fb973f-594f6dc498-xpwgv"]
Sep 16 07:48:29.801841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.798366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-envoy\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.801841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.798452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-token\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.801841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.798488    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-data\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.801841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.798552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-workload-socket\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.801841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.798600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttq2f\" (UniqueName: \"kubernetes.io/projected/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-kube-api-access-ttq2f\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.801841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.798633    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-workload-certs\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.801841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.798681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-podinfo\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.801841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.798742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istiod-ca-cert\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.801841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.798796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-credential-socket\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.899640 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.899604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istiod-ca-cert\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.899811 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.899653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-credential-socket\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.899811 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.899683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-envoy\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.899811 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.899707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-token\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.899811 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.899729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-data\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.899811 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.899767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-workload-socket\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.899811 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.899804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttq2f\" (UniqueName: \"kubernetes.io/projected/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-kube-api-access-ttq2f\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.900187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.899867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-workload-certs\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.900187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.899911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-podinfo\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.900187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.900130    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-workload-socket\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.900187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.900178    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-credential-socket\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.900380 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.900317    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-data\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.900380 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.900351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-workload-certs\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.900498 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.900421    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istiod-ca-cert\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.902332 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.902314    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-envoy\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.902462 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.902446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-podinfo\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.909812 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.909774    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-token\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.910464 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.910443    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttq2f\" (UniqueName: \"kubernetes.io/projected/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-kube-api-access-ttq2f\") pod \"e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") " pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:29.986681 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:29.986604    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:30.272646 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:30.272619    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"]
Sep 16 07:48:30.273969 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:48:30.273943    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb1cdcc_23a7_4a80_8dd3_8f3ed2a908d2.slice/crio-6da8385318e1f058b7054de848c3c446019e7eb3ed2b011c7dd4e32f1c1232bb WatchSource:0}: Error finding container 6da8385318e1f058b7054de848c3c446019e7eb3ed2b011c7dd4e32f1c1232bb: Status 404 returned error can't find the container with id 6da8385318e1f058b7054de848c3c446019e7eb3ed2b011c7dd4e32f1c1232bb
Sep 16 07:48:30.276253 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:30.276220    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:48:30.276335 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:30.276284    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:48:30.276335 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:30.276313    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:48:30.734601 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:30.734562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz" event={"ID":"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2","Type":"ContainerStarted","Data":"a6f8ea36b78724a57f752bc80050f878602c64fb10f754fcc34527e6d26127dd"}
Sep 16 07:48:30.735062 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:30.734621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz" event={"ID":"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2","Type":"ContainerStarted","Data":"6da8385318e1f058b7054de848c3c446019e7eb3ed2b011c7dd4e32f1c1232bb"}
Sep 16 07:48:30.757146 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:30.757091    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz" podStartSLOduration=1.757076815 podStartE2EDuration="1.757076815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:48:30.755313831 +0000 UTC m=+871.681925984" watchObservedRunningTime="2026-09-16 07:48:30.757076815 +0000 UTC m=+871.683688967"
Sep 16 07:48:30.987251 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:30.987165    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:31.684380 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:31.684341    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2025360b-5f2d-4a5c-9448-0d57d5af6c05" path="/var/lib/kubelet/pods/2025360b-5f2d-4a5c-9448-0d57d5af6c05/volumes"
Sep 16 07:48:31.684790 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:31.684771    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a57a5b27-a825-406b-9bb2-30ac477c18a5" path="/var/lib/kubelet/pods/a57a5b27-a825-406b-9bb2-30ac477c18a5/volumes"
Sep 16 07:48:31.992100 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:31.992014    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:32.490802 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:32.490770    2581 patch_prober.go:28] interesting pod/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 07:48:32.491027 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:32.490848    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d" podUID="41c1d84b-657f-46b5-a767-97a20bbe614f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 07:48:32.746508 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:32.746423    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:32.747348 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:32.747323    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:34.188644 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.188621    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:48:34.345622 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.345520    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-envoy\") pod \"41c1d84b-657f-46b5-a767-97a20bbe614f\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") "
Sep 16 07:48:34.345622 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.345567    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-data\") pod \"41c1d84b-657f-46b5-a767-97a20bbe614f\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") "
Sep 16 07:48:34.345622 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.345600    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9bbs\" (UniqueName: \"kubernetes.io/projected/41c1d84b-657f-46b5-a767-97a20bbe614f-kube-api-access-v9bbs\") pod \"41c1d84b-657f-46b5-a767-97a20bbe614f\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") "
Sep 16 07:48:34.345907 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.345632    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-workload-certs\") pod \"41c1d84b-657f-46b5-a767-97a20bbe614f\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") "
Sep 16 07:48:34.345907 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.345707    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-token\") pod \"41c1d84b-657f-46b5-a767-97a20bbe614f\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") "
Sep 16 07:48:34.345907 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.345784    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-podinfo\") pod \"41c1d84b-657f-46b5-a767-97a20bbe614f\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") "
Sep 16 07:48:34.345907 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.345884    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-workload-socket\") pod \"41c1d84b-657f-46b5-a767-97a20bbe614f\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") "
Sep 16 07:48:34.346126 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.345923    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-credential-socket\") pod \"41c1d84b-657f-46b5-a767-97a20bbe614f\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") "
Sep 16 07:48:34.346126 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.345953    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c1d84b-657f-46b5-a767-97a20bbe614f-istiod-ca-cert\") pod \"41c1d84b-657f-46b5-a767-97a20bbe614f\" (UID: \"41c1d84b-657f-46b5-a767-97a20bbe614f\") "
Sep 16 07:48:34.346359 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.345875    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-data" (OuterVolumeSpecName: "istio-data") pod "41c1d84b-657f-46b5-a767-97a20bbe614f" (UID: "41c1d84b-657f-46b5-a767-97a20bbe614f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:48:34.346557 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.346055    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "41c1d84b-657f-46b5-a767-97a20bbe614f" (UID: "41c1d84b-657f-46b5-a767-97a20bbe614f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:48:34.346689 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.346186    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "41c1d84b-657f-46b5-a767-97a20bbe614f" (UID: "41c1d84b-657f-46b5-a767-97a20bbe614f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:48:34.346800 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.346308    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "41c1d84b-657f-46b5-a767-97a20bbe614f" (UID: "41c1d84b-657f-46b5-a767-97a20bbe614f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:48:34.346943 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.346588    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c1d84b-657f-46b5-a767-97a20bbe614f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "41c1d84b-657f-46b5-a767-97a20bbe614f" (UID: "41c1d84b-657f-46b5-a767-97a20bbe614f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:48:34.348787 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.348749    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c1d84b-657f-46b5-a767-97a20bbe614f-kube-api-access-v9bbs" (OuterVolumeSpecName: "kube-api-access-v9bbs") pod "41c1d84b-657f-46b5-a767-97a20bbe614f" (UID: "41c1d84b-657f-46b5-a767-97a20bbe614f"). InnerVolumeSpecName "kube-api-access-v9bbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:48:34.348787 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.348778    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-token" (OuterVolumeSpecName: "istio-token") pod "41c1d84b-657f-46b5-a767-97a20bbe614f" (UID: "41c1d84b-657f-46b5-a767-97a20bbe614f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:48:34.349122 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.349102    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "41c1d84b-657f-46b5-a767-97a20bbe614f" (UID: "41c1d84b-657f-46b5-a767-97a20bbe614f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:48:34.350410 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.350389    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "41c1d84b-657f-46b5-a767-97a20bbe614f" (UID: "41c1d84b-657f-46b5-a767-97a20bbe614f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 07:48:34.446863 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.446809    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:48:34.446863 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.446862    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:48:34.447104 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.446873    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c1d84b-657f-46b5-a767-97a20bbe614f-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:48:34.447104 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.446883    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:48:34.447104 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.446890    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:48:34.447104 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.446898    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9bbs\" (UniqueName: \"kubernetes.io/projected/41c1d84b-657f-46b5-a767-97a20bbe614f-kube-api-access-v9bbs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:48:34.447104 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.446906    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c1d84b-657f-46b5-a767-97a20bbe614f-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:48:34.447104 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.446913    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:48:34.447104 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.446921    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c1d84b-657f-46b5-a767-97a20bbe614f-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:48:34.759025 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.758994    2581 generic.go:356] "Generic (PLEG): container finished" podID="41c1d84b-657f-46b5-a767-97a20bbe614f" containerID="c73d0dcde01c9939895c8386fbdf6336d3e3e6039bf7066f338af128e54c5182" exitCode=0
Sep 16 07:48:34.759212 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.759039    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"
Sep 16 07:48:34.759212 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.759075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d" event={"ID":"41c1d84b-657f-46b5-a767-97a20bbe614f","Type":"ContainerDied","Data":"c73d0dcde01c9939895c8386fbdf6336d3e3e6039bf7066f338af128e54c5182"}
Sep 16 07:48:34.759212 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.759137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d" event={"ID":"41c1d84b-657f-46b5-a767-97a20bbe614f","Type":"ContainerDied","Data":"08054d3f92ac1bd7ca5acec646ad9917429598f3f2575b72192a9effb6c0feec"}
Sep 16 07:48:34.759212 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.759169    2581 scope.go:117] "RemoveContainer" containerID="c73d0dcde01c9939895c8386fbdf6336d3e3e6039bf7066f338af128e54c5182"
Sep 16 07:48:34.769338 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.769320    2581 scope.go:117] "RemoveContainer" containerID="2c4c1d97bb9458dffd83d489bbdf8d7f1eaa8710f4c3abd65a824f889f2adee7"
Sep 16 07:48:34.778016 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.777999    2581 scope.go:117] "RemoveContainer" containerID="c73d0dcde01c9939895c8386fbdf6336d3e3e6039bf7066f338af128e54c5182"
Sep 16 07:48:34.778295 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:48:34.778276    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c73d0dcde01c9939895c8386fbdf6336d3e3e6039bf7066f338af128e54c5182\": container with ID starting with c73d0dcde01c9939895c8386fbdf6336d3e3e6039bf7066f338af128e54c5182 not found: ID does not exist" containerID="c73d0dcde01c9939895c8386fbdf6336d3e3e6039bf7066f338af128e54c5182"
Sep 16 07:48:34.778364 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.778308    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c73d0dcde01c9939895c8386fbdf6336d3e3e6039bf7066f338af128e54c5182"} err="failed to get container status \"c73d0dcde01c9939895c8386fbdf6336d3e3e6039bf7066f338af128e54c5182\": rpc error: code = NotFound desc = could not find container \"c73d0dcde01c9939895c8386fbdf6336d3e3e6039bf7066f338af128e54c5182\": container with ID starting with c73d0dcde01c9939895c8386fbdf6336d3e3e6039bf7066f338af128e54c5182 not found: ID does not exist"
Sep 16 07:48:34.778364 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.778333    2581 scope.go:117] "RemoveContainer" containerID="2c4c1d97bb9458dffd83d489bbdf8d7f1eaa8710f4c3abd65a824f889f2adee7"
Sep 16 07:48:34.778601 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:48:34.778583    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c4c1d97bb9458dffd83d489bbdf8d7f1eaa8710f4c3abd65a824f889f2adee7\": container with ID starting with 2c4c1d97bb9458dffd83d489bbdf8d7f1eaa8710f4c3abd65a824f889f2adee7 not found: ID does not exist" containerID="2c4c1d97bb9458dffd83d489bbdf8d7f1eaa8710f4c3abd65a824f889f2adee7"
Sep 16 07:48:34.778644 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.778606    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c4c1d97bb9458dffd83d489bbdf8d7f1eaa8710f4c3abd65a824f889f2adee7"} err="failed to get container status \"2c4c1d97bb9458dffd83d489bbdf8d7f1eaa8710f4c3abd65a824f889f2adee7\": rpc error: code = NotFound desc = could not find container \"2c4c1d97bb9458dffd83d489bbdf8d7f1eaa8710f4c3abd65a824f889f2adee7\": container with ID starting with 2c4c1d97bb9458dffd83d489bbdf8d7f1eaa8710f4c3abd65a824f889f2adee7 not found: ID does not exist"
Sep 16 07:48:34.800035 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.800015    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"]
Sep 16 07:48:34.822958 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.822932    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"]
Sep 16 07:48:34.823340 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.823327    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c1d84b-657f-46b5-a767-97a20bbe614f" containerName="istio-proxy"
Sep 16 07:48:34.823383 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.823344    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c1d84b-657f-46b5-a767-97a20bbe614f" containerName="istio-proxy"
Sep 16 07:48:34.823383 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.823362    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c1d84b-657f-46b5-a767-97a20bbe614f" containerName="istio-proxy"
Sep 16 07:48:34.823383 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.823368    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c1d84b-657f-46b5-a767-97a20bbe614f" containerName="istio-proxy"
Sep 16 07:48:34.823478 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.823444    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="41c1d84b-657f-46b5-a767-97a20bbe614f" containerName="istio-proxy"
Sep 16 07:48:34.823478 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.823453    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="41c1d84b-657f-46b5-a767-97a20bbe614f" containerName="istio-proxy"
Sep 16 07:48:34.826198 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.826181    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e5fb973f-gw-openshift-default-59c6b47648-fwd7d"]
Sep 16 07:48:34.826268 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.826258    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"
Sep 16 07:48:34.829481 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.829465    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4d9f69e4\""
Sep 16 07:48:34.841906 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.841881    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"]
Sep 16 07:48:34.844089 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.844070    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"]
Sep 16 07:48:34.846275 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.846257    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"
Sep 16 07:48:34.848891 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.848873    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4d9f69e4-dockercfg-cfmqt\""
Sep 16 07:48:34.860301 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.860281    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"]
Sep 16 07:48:34.955639 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.955166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d89136f-ab35-4211-8690-4a5788162926-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk\" (UID: \"9d89136f-ab35-4211-8690-4a5788162926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"
Sep 16 07:48:34.955639 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.955279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee498ad-a49a-428f-810c-663b04899a21-plugins-config-volume\") pod \"payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m\" (UID: \"eee498ad-a49a-428f-810c-663b04899a21\") " pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"
Sep 16 07:48:34.955639 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.955319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp9st\" (UniqueName: \"kubernetes.io/projected/9d89136f-ab35-4211-8690-4a5788162926-kube-api-access-wp9st\") pod \"payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk\" (UID: \"9d89136f-ab35-4211-8690-4a5788162926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"
Sep 16 07:48:34.955639 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:34.955377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klrjm\" (UniqueName: \"kubernetes.io/projected/eee498ad-a49a-428f-810c-663b04899a21-kube-api-access-klrjm\") pod \"payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m\" (UID: \"eee498ad-a49a-428f-810c-663b04899a21\") " pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"
Sep 16 07:48:35.055317 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.055235    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"]
Sep 16 07:48:35.056313 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.056286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee498ad-a49a-428f-810c-663b04899a21-plugins-config-volume\") pod \"payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m\" (UID: \"eee498ad-a49a-428f-810c-663b04899a21\") " pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"
Sep 16 07:48:35.056469 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.056324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp9st\" (UniqueName: \"kubernetes.io/projected/9d89136f-ab35-4211-8690-4a5788162926-kube-api-access-wp9st\") pod \"payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk\" (UID: \"9d89136f-ab35-4211-8690-4a5788162926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"
Sep 16 07:48:35.056469 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.056376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klrjm\" (UniqueName: \"kubernetes.io/projected/eee498ad-a49a-428f-810c-663b04899a21-kube-api-access-klrjm\") pod \"payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m\" (UID: \"eee498ad-a49a-428f-810c-663b04899a21\") " pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"
Sep 16 07:48:35.056584 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.056472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d89136f-ab35-4211-8690-4a5788162926-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk\" (UID: \"9d89136f-ab35-4211-8690-4a5788162926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"
Sep 16 07:48:35.057219 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.057197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee498ad-a49a-428f-810c-663b04899a21-plugins-config-volume\") pod \"payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m\" (UID: \"eee498ad-a49a-428f-810c-663b04899a21\") " pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"
Sep 16 07:48:35.057319 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.057300    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d89136f-ab35-4211-8690-4a5788162926-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk\" (UID: \"9d89136f-ab35-4211-8690-4a5788162926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"
Sep 16 07:48:35.058037 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.058018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.060519 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.060492    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b60434\"/\"kube-root-ca.crt\""
Sep 16 07:48:35.060676 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.060650    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b60434\"/\"openshift-service-ca.crt\""
Sep 16 07:48:35.060676 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.060667    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b60434\"/\"autores-model-b60434-kserve-self-signed-certs\""
Sep 16 07:48:35.060882 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.060722    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b60434\"/\"default-dockercfg-bf99w\""
Sep 16 07:48:35.064359 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.064339    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp9st\" (UniqueName: \"kubernetes.io/projected/9d89136f-ab35-4211-8690-4a5788162926-kube-api-access-wp9st\") pod \"payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk\" (UID: \"9d89136f-ab35-4211-8690-4a5788162926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"
Sep 16 07:48:35.064458 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.064364    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klrjm\" (UniqueName: \"kubernetes.io/projected/eee498ad-a49a-428f-810c-663b04899a21-kube-api-access-klrjm\") pod \"payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m\" (UID: \"eee498ad-a49a-428f-810c-663b04899a21\") " pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"
Sep 16 07:48:35.071702 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.071679    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"]
Sep 16 07:48:35.136676 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.136652    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"
Sep 16 07:48:35.155360 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.155337    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"
Sep 16 07:48:35.157159 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.157073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-model-cache\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.157159 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.157113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwmh6\" (UniqueName: \"kubernetes.io/projected/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-kube-api-access-kwmh6\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.157287 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.157247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-home\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.157343 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.157313    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-tmp-dir\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.157402 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.157368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-tls-certs\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.157402 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.157394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-dshm\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.258458 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.258428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-home\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.258863 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.258497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-tmp-dir\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.258863 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.258557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-tls-certs\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.258863 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.258584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-dshm\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.258863 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.258659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-model-cache\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.258863 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.258691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwmh6\" (UniqueName: \"kubernetes.io/projected/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-kube-api-access-kwmh6\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.259128 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.258896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-home\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.259128 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.258949    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-tmp-dir\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.259128 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.258990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-model-cache\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.261001 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.260976    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-dshm\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.261766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.261743    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-tls-certs\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.266608 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.266586    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwmh6\" (UniqueName: \"kubernetes.io/projected/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-kube-api-access-kwmh6\") pod \"autores-model-b60434-kserve-7bb65b97f5-kvhxx\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") " pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.378657 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.378575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:35.408657 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.408624    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"]
Sep 16 07:48:35.410402 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:48:35.410372    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d89136f_ab35_4211_8690_4a5788162926.slice/crio-cc22f08481ac8b17b27bf7a2063fae41903c78f832b782701867ee9c21bd209f WatchSource:0}: Error finding container cc22f08481ac8b17b27bf7a2063fae41903c78f832b782701867ee9c21bd209f: Status 404 returned error can't find the container with id cc22f08481ac8b17b27bf7a2063fae41903c78f832b782701867ee9c21bd209f
Sep 16 07:48:35.453595 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.453565    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"]
Sep 16 07:48:35.463056 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:48:35.463030    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee498ad_a49a_428f_810c_663b04899a21.slice/crio-77f01553b439cb222b1aea640e103ea4d725fe39656aa7fd96b9ba4d06dd9f05 WatchSource:0}: Error finding container 77f01553b439cb222b1aea640e103ea4d725fe39656aa7fd96b9ba4d06dd9f05: Status 404 returned error can't find the container with id 77f01553b439cb222b1aea640e103ea4d725fe39656aa7fd96b9ba4d06dd9f05
Sep 16 07:48:35.656589 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.656420    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"]
Sep 16 07:48:35.658915 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:48:35.658888    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb2f87c8_421c_4f0c_b0ae_8ebfad69370c.slice/crio-5174504bc934a7ab267cd92992f46c9526206ad26bebd948e8bfa5b549122d34 WatchSource:0}: Error finding container 5174504bc934a7ab267cd92992f46c9526206ad26bebd948e8bfa5b549122d34: Status 404 returned error can't find the container with id 5174504bc934a7ab267cd92992f46c9526206ad26bebd948e8bfa5b549122d34
Sep 16 07:48:35.684285 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.684264    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41c1d84b-657f-46b5-a767-97a20bbe614f" path="/var/lib/kubelet/pods/41c1d84b-657f-46b5-a767-97a20bbe614f/volumes"
Sep 16 07:48:35.775741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.775701    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx" event={"ID":"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c","Type":"ContainerStarted","Data":"65d7d3fb15163acb8839398094dd5cf228679383c6d0daedd22304dbcc813a79"}
Sep 16 07:48:35.776034 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.776013    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx" event={"ID":"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c","Type":"ContainerStarted","Data":"5174504bc934a7ab267cd92992f46c9526206ad26bebd948e8bfa5b549122d34"}
Sep 16 07:48:35.777739 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.777710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk" event={"ID":"9d89136f-ab35-4211-8690-4a5788162926","Type":"ContainerStarted","Data":"8a6bb670138c753ab11fa5e380f9e19b753e70eb539f0959c2d60eada8216282"}
Sep 16 07:48:35.777876 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.777795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk" event={"ID":"9d89136f-ab35-4211-8690-4a5788162926","Type":"ContainerStarted","Data":"cc22f08481ac8b17b27bf7a2063fae41903c78f832b782701867ee9c21bd209f"}
Sep 16 07:48:35.777876 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.777838    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"
Sep 16 07:48:35.779493 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.779466    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m" event={"ID":"eee498ad-a49a-428f-810c-663b04899a21","Type":"ContainerStarted","Data":"86071fbc3918341a03425d38de119cf31f527f3f066458a72772f403e3f6e9ed"}
Sep 16 07:48:35.779575 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.779514    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m" event={"ID":"eee498ad-a49a-428f-810c-663b04899a21","Type":"ContainerStarted","Data":"77f01553b439cb222b1aea640e103ea4d725fe39656aa7fd96b9ba4d06dd9f05"}
Sep 16 07:48:35.779575 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.779537    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"
Sep 16 07:48:35.792667 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.792623    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx" podStartSLOduration=0.792610529 podStartE2EDuration="792.610529ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:48:35.79175969 +0000 UTC m=+876.718371847" watchObservedRunningTime="2026-09-16 07:48:35.792610529 +0000 UTC m=+876.719222719"
Sep 16 07:48:35.806178 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.806093    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m" podStartSLOduration=1.8060769319999999 podStartE2EDuration="1.806076932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:48:35.805587905 +0000 UTC m=+876.732200070" watchObservedRunningTime="2026-09-16 07:48:35.806076932 +0000 UTC m=+876.732689085"
Sep 16 07:48:35.821549 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:35.821513    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk" podStartSLOduration=1.8215026060000001 podStartE2EDuration="1.821502606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:48:35.820109953 +0000 UTC m=+876.746722107" watchObservedRunningTime="2026-09-16 07:48:35.821502606 +0000 UTC m=+876.748114799"
Sep 16 07:48:45.379492 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:45.379451    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:45.379911 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:45.379509    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:45.391863 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:45.391815    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:45.847184 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:45.847157    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:48:46.786643 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:46.786615    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"
Sep 16 07:48:46.787047 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:46.786690    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"
Sep 16 07:48:54.885614 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:54.885583    2581 generic.go:356] "Generic (PLEG): container finished" podID="9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" containerID="a6f8ea36b78724a57f752bc80050f878602c64fb10f754fcc34527e6d26127dd" exitCode=0
Sep 16 07:48:54.886055 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:54.885654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz" event={"ID":"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2","Type":"ContainerDied","Data":"a6f8ea36b78724a57f752bc80050f878602c64fb10f754fcc34527e6d26127dd"}
Sep 16 07:48:54.886055 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:54.886020    2581 scope.go:117] "RemoveContainer" containerID="a6f8ea36b78724a57f752bc80050f878602c64fb10f754fcc34527e6d26127dd"
Sep 16 07:48:54.887081 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:54.887055    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:48:54.887139 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:54.887120    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:48:54.887175 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:54.887150    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:48:55.893834 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:55.893788    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz" event={"ID":"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2","Type":"ContainerStarted","Data":"40dc3605e4c5026e2ddcae2e44eb69a2e7fb3b7aac9359508cde96f6a2cd6510"}
Sep 16 07:48:55.987572 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:55.987543    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:48:55.987768 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:55.987744    2581 patch_prober.go:28] interesting pod/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 16 07:48:55.987877 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:55.987800    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz" podUID="9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 16 07:48:56.987771 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:56.987737    2581 patch_prober.go:28] interesting pod/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 16 07:48:56.988174 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:56.987786    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz" podUID="9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 16 07:48:57.987509 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:57.987479    2581 patch_prober.go:28] interesting pod/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 16 07:48:57.987705 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:57.987537    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz" podUID="9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 16 07:48:58.987810 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:58.987772    2581 patch_prober.go:28] interesting pod/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 16 07:48:58.988206 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:58.987857    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz" podUID="9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 16 07:48:59.559596 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:59.559568    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 07:48:59.559596 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:59.559589    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 07:48:59.987180 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:59.987147    2581 patch_prober.go:28] interesting pod/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 16 07:48:59.987343 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:48:59.987202    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz" podUID="9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 16 07:49:01.044744 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:01.044708    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:49:01.045256 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:01.045161    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:49:01.046444 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:01.046411    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:49:04.273642 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.273610    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"]
Sep 16 07:49:04.274042 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.273796    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m" podUID="eee498ad-a49a-428f-810c-663b04899a21" containerName="payload-processing" containerID="cri-o://86071fbc3918341a03425d38de119cf31f527f3f066458a72772f403e3f6e9ed" gracePeriod=30
Sep 16 07:49:04.304024 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.300418    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"]
Sep 16 07:49:04.304024 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.300643    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk" podUID="9d89136f-ab35-4211-8690-4a5788162926" containerName="payload-pre-processing" containerID="cri-o://8a6bb670138c753ab11fa5e380f9e19b753e70eb539f0959c2d60eada8216282" gracePeriod=30
Sep 16 07:49:04.589844 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.588837    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"
Sep 16 07:49:04.621095 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.621071    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"
Sep 16 07:49:04.755798 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.755760    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp9st\" (UniqueName: \"kubernetes.io/projected/9d89136f-ab35-4211-8690-4a5788162926-kube-api-access-wp9st\") pod \"9d89136f-ab35-4211-8690-4a5788162926\" (UID: \"9d89136f-ab35-4211-8690-4a5788162926\") "
Sep 16 07:49:04.755798 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.755807    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d89136f-ab35-4211-8690-4a5788162926-plugins-config-volume\") pod \"9d89136f-ab35-4211-8690-4a5788162926\" (UID: \"9d89136f-ab35-4211-8690-4a5788162926\") "
Sep 16 07:49:04.756065 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.755961    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee498ad-a49a-428f-810c-663b04899a21-plugins-config-volume\") pod \"eee498ad-a49a-428f-810c-663b04899a21\" (UID: \"eee498ad-a49a-428f-810c-663b04899a21\") "
Sep 16 07:49:04.756136 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.756071    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klrjm\" (UniqueName: \"kubernetes.io/projected/eee498ad-a49a-428f-810c-663b04899a21-kube-api-access-klrjm\") pod \"eee498ad-a49a-428f-810c-663b04899a21\" (UID: \"eee498ad-a49a-428f-810c-663b04899a21\") "
Sep 16 07:49:04.756215 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.756193    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d89136f-ab35-4211-8690-4a5788162926-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d89136f-ab35-4211-8690-4a5788162926" (UID: "9d89136f-ab35-4211-8690-4a5788162926"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:49:04.756308 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.756287    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee498ad-a49a-428f-810c-663b04899a21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eee498ad-a49a-428f-810c-663b04899a21" (UID: "eee498ad-a49a-428f-810c-663b04899a21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:49:04.756404 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.756390    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d89136f-ab35-4211-8690-4a5788162926-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:04.756465 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.756405    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eee498ad-a49a-428f-810c-663b04899a21-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:04.757918 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.757898    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d89136f-ab35-4211-8690-4a5788162926-kube-api-access-wp9st" (OuterVolumeSpecName: "kube-api-access-wp9st") pod "9d89136f-ab35-4211-8690-4a5788162926" (UID: "9d89136f-ab35-4211-8690-4a5788162926"). InnerVolumeSpecName "kube-api-access-wp9st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:49:04.758014 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.757949    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee498ad-a49a-428f-810c-663b04899a21-kube-api-access-klrjm" (OuterVolumeSpecName: "kube-api-access-klrjm") pod "eee498ad-a49a-428f-810c-663b04899a21" (UID: "eee498ad-a49a-428f-810c-663b04899a21"). InnerVolumeSpecName "kube-api-access-klrjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:49:04.857270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.857213    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klrjm\" (UniqueName: \"kubernetes.io/projected/eee498ad-a49a-428f-810c-663b04899a21-kube-api-access-klrjm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:04.857270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.857234    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp9st\" (UniqueName: \"kubernetes.io/projected/9d89136f-ab35-4211-8690-4a5788162926-kube-api-access-wp9st\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:04.944549 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.944523    2581 generic.go:356] "Generic (PLEG): container finished" podID="eee498ad-a49a-428f-810c-663b04899a21" containerID="86071fbc3918341a03425d38de119cf31f527f3f066458a72772f403e3f6e9ed" exitCode=0
Sep 16 07:49:04.949856 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.944573    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"
Sep 16 07:49:04.949856 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.944592    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m" event={"ID":"eee498ad-a49a-428f-810c-663b04899a21","Type":"ContainerDied","Data":"86071fbc3918341a03425d38de119cf31f527f3f066458a72772f403e3f6e9ed"}
Sep 16 07:49:04.949856 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.945608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m" event={"ID":"eee498ad-a49a-428f-810c-663b04899a21","Type":"ContainerDied","Data":"77f01553b439cb222b1aea640e103ea4d725fe39656aa7fd96b9ba4d06dd9f05"}
Sep 16 07:49:04.949856 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.945642    2581 scope.go:117] "RemoveContainer" containerID="86071fbc3918341a03425d38de119cf31f527f3f066458a72772f403e3f6e9ed"
Sep 16 07:49:04.963557 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.963118    2581 generic.go:356] "Generic (PLEG): container finished" podID="9d89136f-ab35-4211-8690-4a5788162926" containerID="8a6bb670138c753ab11fa5e380f9e19b753e70eb539f0959c2d60eada8216282" exitCode=0
Sep 16 07:49:04.963557 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.963197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk" event={"ID":"9d89136f-ab35-4211-8690-4a5788162926","Type":"ContainerDied","Data":"8a6bb670138c753ab11fa5e380f9e19b753e70eb539f0959c2d60eada8216282"}
Sep 16 07:49:04.963557 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.963243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk" event={"ID":"9d89136f-ab35-4211-8690-4a5788162926","Type":"ContainerDied","Data":"cc22f08481ac8b17b27bf7a2063fae41903c78f832b782701867ee9c21bd209f"}
Sep 16 07:49:04.963557 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.963315    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"
Sep 16 07:49:04.973245 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.973219    2581 scope.go:117] "RemoveContainer" containerID="86071fbc3918341a03425d38de119cf31f527f3f066458a72772f403e3f6e9ed"
Sep 16 07:49:04.973583 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:49:04.973551    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86071fbc3918341a03425d38de119cf31f527f3f066458a72772f403e3f6e9ed\": container with ID starting with 86071fbc3918341a03425d38de119cf31f527f3f066458a72772f403e3f6e9ed not found: ID does not exist" containerID="86071fbc3918341a03425d38de119cf31f527f3f066458a72772f403e3f6e9ed"
Sep 16 07:49:04.973684 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.973592    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86071fbc3918341a03425d38de119cf31f527f3f066458a72772f403e3f6e9ed"} err="failed to get container status \"86071fbc3918341a03425d38de119cf31f527f3f066458a72772f403e3f6e9ed\": rpc error: code = NotFound desc = could not find container \"86071fbc3918341a03425d38de119cf31f527f3f066458a72772f403e3f6e9ed\": container with ID starting with 86071fbc3918341a03425d38de119cf31f527f3f066458a72772f403e3f6e9ed not found: ID does not exist"
Sep 16 07:49:04.973684 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.973610    2581 scope.go:117] "RemoveContainer" containerID="8a6bb670138c753ab11fa5e380f9e19b753e70eb539f0959c2d60eada8216282"
Sep 16 07:49:04.982140 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.982119    2581 scope.go:117] "RemoveContainer" containerID="8a6bb670138c753ab11fa5e380f9e19b753e70eb539f0959c2d60eada8216282"
Sep 16 07:49:04.982415 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:49:04.982392    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6bb670138c753ab11fa5e380f9e19b753e70eb539f0959c2d60eada8216282\": container with ID starting with 8a6bb670138c753ab11fa5e380f9e19b753e70eb539f0959c2d60eada8216282 not found: ID does not exist" containerID="8a6bb670138c753ab11fa5e380f9e19b753e70eb539f0959c2d60eada8216282"
Sep 16 07:49:04.982505 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.982419    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6bb670138c753ab11fa5e380f9e19b753e70eb539f0959c2d60eada8216282"} err="failed to get container status \"8a6bb670138c753ab11fa5e380f9e19b753e70eb539f0959c2d60eada8216282\": rpc error: code = NotFound desc = could not find container \"8a6bb670138c753ab11fa5e380f9e19b753e70eb539f0959c2d60eada8216282\": container with ID starting with 8a6bb670138c753ab11fa5e380f9e19b753e70eb539f0959c2d60eada8216282 not found: ID does not exist"
Sep 16 07:49:04.982738 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.982720    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"]
Sep 16 07:49:04.989579 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:04.989558    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4d9f69e4-6c774fd665-nvf5m"]
Sep 16 07:49:05.010524 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:05.010497    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"]
Sep 16 07:49:05.015643 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:05.015621    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4d9f69e4-86b4b6c566-h6kfk"]
Sep 16 07:49:05.623944 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:05.623870    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"]
Sep 16 07:49:05.624347 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:05.624096    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz" podUID="9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" containerName="istio-proxy" containerID="cri-o://40dc3605e4c5026e2ddcae2e44eb69a2e7fb3b7aac9359508cde96f6a2cd6510" gracePeriod=30
Sep 16 07:49:05.684314 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:05.684281    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d89136f-ab35-4211-8690-4a5788162926" path="/var/lib/kubelet/pods/9d89136f-ab35-4211-8690-4a5788162926/volumes"
Sep 16 07:49:05.684651 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:05.684638    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eee498ad-a49a-428f-810c-663b04899a21" path="/var/lib/kubelet/pods/eee498ad-a49a-428f-810c-663b04899a21/volumes"
Sep 16 07:49:06.338343 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.338239    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"]
Sep 16 07:49:06.338887 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.338863    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee498ad-a49a-428f-810c-663b04899a21" containerName="payload-processing"
Sep 16 07:49:06.338887 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.338885    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee498ad-a49a-428f-810c-663b04899a21" containerName="payload-processing"
Sep 16 07:49:06.339085 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.338901    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d89136f-ab35-4211-8690-4a5788162926" containerName="payload-pre-processing"
Sep 16 07:49:06.339085 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.338906    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d89136f-ab35-4211-8690-4a5788162926" containerName="payload-pre-processing"
Sep 16 07:49:06.339085 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.338992    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eee498ad-a49a-428f-810c-663b04899a21" containerName="payload-processing"
Sep 16 07:49:06.339085 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.339006    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d89136f-ab35-4211-8690-4a5788162926" containerName="payload-pre-processing"
Sep 16 07:49:06.343786 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.343764    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.346643 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.346624    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1892c35b-openshift-default-dockercfg-sfz6q\""
Sep 16 07:49:06.357126 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.357102    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"]
Sep 16 07:49:06.472694 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.472122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj57f\" (UniqueName: \"kubernetes.io/projected/2d21ce7e-d3a1-4286-ab95-ed813cd66043-kube-api-access-cj57f\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.472694 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.472252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-data\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.472694 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.472365    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-workload-socket\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.472694 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.472397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-credential-socket\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.472694 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.472431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-envoy\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.472694 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.472469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-podinfo\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.472694 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.472495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-token\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.472694 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.472521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istiod-ca-cert\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.472694 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.472547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-workload-certs\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.573142 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.573112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj57f\" (UniqueName: \"kubernetes.io/projected/2d21ce7e-d3a1-4286-ab95-ed813cd66043-kube-api-access-cj57f\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.573344 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.573180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-data\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.573344 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.573235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-workload-socket\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.573344 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.573254    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-credential-socket\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.573344 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.573274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-envoy\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.573344 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.573304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-podinfo\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.573344 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.573328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-token\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.573602 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.573351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istiod-ca-cert\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.573602 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.573377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-workload-certs\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.573985 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.573912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-data\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.573985 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.573936    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-workload-certs\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.573985 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.573974    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-credential-socket\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.574236 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.574136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-workload-socket\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.574396 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.574367    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istiod-ca-cert\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.575879 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.575853    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-envoy\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.576086 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.576067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-podinfo\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.581275 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.581254    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj57f\" (UniqueName: \"kubernetes.io/projected/2d21ce7e-d3a1-4286-ab95-ed813cd66043-kube-api-access-cj57f\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.581364 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.581334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-token\") pod \"e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") " pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.656994 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.656972    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:06.940833 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.940788    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"]
Sep 16 07:49:06.942257 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:49:06.942234    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d21ce7e_d3a1_4286_ab95_ed813cd66043.slice/crio-cc72e586a0556d7ff00614072fec56a5196f1396ef5c44ff8b424d44b2c6fc2c WatchSource:0}: Error finding container cc72e586a0556d7ff00614072fec56a5196f1396ef5c44ff8b424d44b2c6fc2c: Status 404 returned error can't find the container with id cc72e586a0556d7ff00614072fec56a5196f1396ef5c44ff8b424d44b2c6fc2c
Sep 16 07:49:06.944718 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.944675    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:49:06.944799 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.944771    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:49:06.944869 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.944847    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:49:06.976301 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:06.976270    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk" event={"ID":"2d21ce7e-d3a1-4286-ab95-ed813cd66043","Type":"ContainerStarted","Data":"cc72e586a0556d7ff00614072fec56a5196f1396ef5c44ff8b424d44b2c6fc2c"}
Sep 16 07:49:07.982560 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:07.982526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk" event={"ID":"2d21ce7e-d3a1-4286-ab95-ed813cd66043","Type":"ContainerStarted","Data":"81e19f4b371bd3859dee57dde4b0e3f3fc8b08d1e54144799f63a0aa51a77750"}
Sep 16 07:49:08.005206 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.005153    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk" podStartSLOduration=2.005139399 podStartE2EDuration="2.005139399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:49:08.001533117 +0000 UTC m=+908.928145270" watchObservedRunningTime="2026-09-16 07:49:08.005139399 +0000 UTC m=+908.931751551"
Sep 16 07:49:08.221934 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.221899    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b60434-pkhvr"]
Sep 16 07:49:08.225734 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.225717    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b60434-pkhvr"
Sep 16 07:49:08.228264 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.228241    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\""
Sep 16 07:49:08.228360 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.228279    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 07:49:08.233875 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.233710    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b60434-pkhvr"]
Sep 16 07:49:08.392399 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.392364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ec391f-fd5e-490b-ad03-a3ca3d09f94f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b60434-pkhvr\" (UID: \"64ec391f-fd5e-490b-ad03-a3ca3d09f94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b60434-pkhvr"
Sep 16 07:49:08.493778 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.493696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ec391f-fd5e-490b-ad03-a3ca3d09f94f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b60434-pkhvr\" (UID: \"64ec391f-fd5e-490b-ad03-a3ca3d09f94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b60434-pkhvr"
Sep 16 07:49:08.494319 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.494301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ec391f-fd5e-490b-ad03-a3ca3d09f94f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b60434-pkhvr\" (UID: \"64ec391f-fd5e-490b-ad03-a3ca3d09f94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b60434-pkhvr"
Sep 16 07:49:08.540449 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.540422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b60434-pkhvr"
Sep 16 07:49:08.657736 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.657698    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:08.662571 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.662550    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:08.808425 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.808399    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b60434-pkhvr"]
Sep 16 07:49:08.810680 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:49:08.810626    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ec391f_fd5e_490b_ad03_a3ca3d09f94f.slice/crio-c0553ec076dcd31d5d08af3f7c021fc1bebc5758707e522057ec9ace1ded6e52 WatchSource:0}: Error finding container c0553ec076dcd31d5d08af3f7c021fc1bebc5758707e522057ec9ace1ded6e52: Status 404 returned error can't find the container with id c0553ec076dcd31d5d08af3f7c021fc1bebc5758707e522057ec9ace1ded6e52
Sep 16 07:49:08.988831 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.988788    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b60434-pkhvr" event={"ID":"64ec391f-fd5e-490b-ad03-a3ca3d09f94f","Type":"ContainerStarted","Data":"c0553ec076dcd31d5d08af3f7c021fc1bebc5758707e522057ec9ace1ded6e52"}
Sep 16 07:49:08.989237 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.989147    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:08.989921 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:08.989902    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:09.994860 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:09.994809    2581 generic.go:356] "Generic (PLEG): container finished" podID="64ec391f-fd5e-490b-ad03-a3ca3d09f94f" containerID="cd4d6418f782f6442b43379b986c9cfba25a4d21879669698521d174a667935c" exitCode=0
Sep 16 07:49:09.995266 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:09.994897    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b60434-pkhvr" event={"ID":"64ec391f-fd5e-490b-ad03-a3ca3d09f94f","Type":"ContainerDied","Data":"cd4d6418f782f6442b43379b986c9cfba25a4d21879669698521d174a667935c"}
Sep 16 07:49:10.897927 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:10.897866    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:49:11.001992 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.001960    2581 generic.go:356] "Generic (PLEG): container finished" podID="9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" containerID="40dc3605e4c5026e2ddcae2e44eb69a2e7fb3b7aac9359508cde96f6a2cd6510" exitCode=0
Sep 16 07:49:11.002358 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.002003    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"
Sep 16 07:49:11.002358 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.002044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz" event={"ID":"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2","Type":"ContainerDied","Data":"40dc3605e4c5026e2ddcae2e44eb69a2e7fb3b7aac9359508cde96f6a2cd6510"}
Sep 16 07:49:11.002358 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.002088    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz" event={"ID":"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2","Type":"ContainerDied","Data":"6da8385318e1f058b7054de848c3c446019e7eb3ed2b011c7dd4e32f1c1232bb"}
Sep 16 07:49:11.002358 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.002109    2581 scope.go:117] "RemoveContainer" containerID="40dc3605e4c5026e2ddcae2e44eb69a2e7fb3b7aac9359508cde96f6a2cd6510"
Sep 16 07:49:11.011731 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.011704    2581 scope.go:117] "RemoveContainer" containerID="a6f8ea36b78724a57f752bc80050f878602c64fb10f754fcc34527e6d26127dd"
Sep 16 07:49:11.018170 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.018133    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-podinfo\") pod \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") "
Sep 16 07:49:11.018292 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.018175    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-workload-socket\") pod \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") "
Sep 16 07:49:11.018292 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.018210    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-token\") pod \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") "
Sep 16 07:49:11.018292 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.018273    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-workload-certs\") pod \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") "
Sep 16 07:49:11.018461 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.018303    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istiod-ca-cert\") pod \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") "
Sep 16 07:49:11.018461 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.018382    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-credential-socket\") pod \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") "
Sep 16 07:49:11.018461 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.018412    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttq2f\" (UniqueName: \"kubernetes.io/projected/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-kube-api-access-ttq2f\") pod \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") "
Sep 16 07:49:11.018620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.018462    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-data\") pod \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") "
Sep 16 07:49:11.018620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.018491    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" (UID: "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:49:11.018763 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.018737    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" (UID: "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:49:11.018908 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.018586    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" (UID: "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:49:11.018908 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.018872    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" (UID: "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:49:11.019031 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.018525    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-envoy\") pod \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\" (UID: \"9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2\") "
Sep 16 07:49:11.019340 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.019317    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:11.019432 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.019345    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:11.019432 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.019361    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:11.019432 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.019374    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:11.019715 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.019677    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-data" (OuterVolumeSpecName: "istio-data") pod "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" (UID: "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:49:11.020871 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.020452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" (UID: "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:49:11.021346 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.021304    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-kube-api-access-ttq2f" (OuterVolumeSpecName: "kube-api-access-ttq2f") pod "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" (UID: "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2"). InnerVolumeSpecName "kube-api-access-ttq2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:49:11.022046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.022024    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-token" (OuterVolumeSpecName: "istio-token") pod "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" (UID: "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:49:11.022333 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.022256    2581 scope.go:117] "RemoveContainer" containerID="40dc3605e4c5026e2ddcae2e44eb69a2e7fb3b7aac9359508cde96f6a2cd6510"
Sep 16 07:49:11.022333 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.022298    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" (UID: "9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 07:49:11.022533 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:49:11.022512    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40dc3605e4c5026e2ddcae2e44eb69a2e7fb3b7aac9359508cde96f6a2cd6510\": container with ID starting with 40dc3605e4c5026e2ddcae2e44eb69a2e7fb3b7aac9359508cde96f6a2cd6510 not found: ID does not exist" containerID="40dc3605e4c5026e2ddcae2e44eb69a2e7fb3b7aac9359508cde96f6a2cd6510"
Sep 16 07:49:11.022600 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.022546    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40dc3605e4c5026e2ddcae2e44eb69a2e7fb3b7aac9359508cde96f6a2cd6510"} err="failed to get container status \"40dc3605e4c5026e2ddcae2e44eb69a2e7fb3b7aac9359508cde96f6a2cd6510\": rpc error: code = NotFound desc = could not find container \"40dc3605e4c5026e2ddcae2e44eb69a2e7fb3b7aac9359508cde96f6a2cd6510\": container with ID starting with 40dc3605e4c5026e2ddcae2e44eb69a2e7fb3b7aac9359508cde96f6a2cd6510 not found: ID does not exist"
Sep 16 07:49:11.022600 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.022571    2581 scope.go:117] "RemoveContainer" containerID="a6f8ea36b78724a57f752bc80050f878602c64fb10f754fcc34527e6d26127dd"
Sep 16 07:49:11.022892 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:49:11.022871    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6f8ea36b78724a57f752bc80050f878602c64fb10f754fcc34527e6d26127dd\": container with ID starting with a6f8ea36b78724a57f752bc80050f878602c64fb10f754fcc34527e6d26127dd not found: ID does not exist" containerID="a6f8ea36b78724a57f752bc80050f878602c64fb10f754fcc34527e6d26127dd"
Sep 16 07:49:11.022954 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.022898    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f8ea36b78724a57f752bc80050f878602c64fb10f754fcc34527e6d26127dd"} err="failed to get container status \"a6f8ea36b78724a57f752bc80050f878602c64fb10f754fcc34527e6d26127dd\": rpc error: code = NotFound desc = could not find container \"a6f8ea36b78724a57f752bc80050f878602c64fb10f754fcc34527e6d26127dd\": container with ID starting with a6f8ea36b78724a57f752bc80050f878602c64fb10f754fcc34527e6d26127dd not found: ID does not exist"
Sep 16 07:49:11.120007 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.119978    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:11.120007 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.120009    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:11.120219 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.120022    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:11.120219 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.120036    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttq2f\" (UniqueName: \"kubernetes.io/projected/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-kube-api-access-ttq2f\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:11.120219 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.120057    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:11.128369 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.128348    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b60434-pkhvr"
Sep 16 07:49:11.221116 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.221087    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ec391f-fd5e-490b-ad03-a3ca3d09f94f-maas-api-service-ca\") pod \"64ec391f-fd5e-490b-ad03-a3ca3d09f94f\" (UID: \"64ec391f-fd5e-490b-ad03-a3ca3d09f94f\") "
Sep 16 07:49:11.221409 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.221386    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ec391f-fd5e-490b-ad03-a3ca3d09f94f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64ec391f-fd5e-490b-ad03-a3ca3d09f94f" (UID: "64ec391f-fd5e-490b-ad03-a3ca3d09f94f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:49:11.322055 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.322030    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ec391f-fd5e-490b-ad03-a3ca3d09f94f-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:11.334385 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.334360    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"]
Sep 16 07:49:11.341533 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.341507    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4d9f69e4-openshift-default-568846797d-n7hpz"]
Sep 16 07:49:11.465144 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.465117    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"]
Sep 16 07:49:11.465523 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.465511    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" containerName="istio-proxy"
Sep 16 07:49:11.465579 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.465525    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" containerName="istio-proxy"
Sep 16 07:49:11.465579 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.465532    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ec391f-fd5e-490b-ad03-a3ca3d09f94f" containerName="revoke-keys"
Sep 16 07:49:11.465579 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.465538    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ec391f-fd5e-490b-ad03-a3ca3d09f94f" containerName="revoke-keys"
Sep 16 07:49:11.465579 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.465553    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" containerName="istio-proxy"
Sep 16 07:49:11.465579 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.465560    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" containerName="istio-proxy"
Sep 16 07:49:11.465759 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.465619    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" containerName="istio-proxy"
Sep 16 07:49:11.465759 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.465629    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" containerName="istio-proxy"
Sep 16 07:49:11.465759 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.465634    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ec391f-fd5e-490b-ad03-a3ca3d09f94f" containerName="revoke-keys"
Sep 16 07:49:11.468715 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.468699    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"
Sep 16 07:49:11.471532 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.471439    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-1892c35b\""
Sep 16 07:49:11.471532 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.471511    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-1892c35b-dockercfg-tnqgr\""
Sep 16 07:49:11.478794 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.478770    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"]
Sep 16 07:49:11.495010 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.494988    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"]
Sep 16 07:49:11.498380 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.498363    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"
Sep 16 07:49:11.521685 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.521659    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"]
Sep 16 07:49:11.625603 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.625566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2svr5\" (UniqueName: \"kubernetes.io/projected/3370f3a6-ccf5-41e5-9052-bb8dd0308d68-kube-api-access-2svr5\") pod \"payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7\" (UID: \"3370f3a6-ccf5-41e5-9052-bb8dd0308d68\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"
Sep 16 07:49:11.625771 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.625626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293fa14a-7b22-4b04-b747-e356dc56ee36-plugins-config-volume\") pod \"payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq\" (UID: \"293fa14a-7b22-4b04-b747-e356dc56ee36\") " pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"
Sep 16 07:49:11.625771 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.625748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3370f3a6-ccf5-41e5-9052-bb8dd0308d68-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7\" (UID: \"3370f3a6-ccf5-41e5-9052-bb8dd0308d68\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"
Sep 16 07:49:11.625885 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.625802    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqt5\" (UniqueName: \"kubernetes.io/projected/293fa14a-7b22-4b04-b747-e356dc56ee36-kube-api-access-ptqt5\") pod \"payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq\" (UID: \"293fa14a-7b22-4b04-b747-e356dc56ee36\") " pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"
Sep 16 07:49:11.684056 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.684022    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2" path="/var/lib/kubelet/pods/9eb1cdcc-23a7-4a80-8dd3-8f3ed2a908d2/volumes"
Sep 16 07:49:11.726792 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.726733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2svr5\" (UniqueName: \"kubernetes.io/projected/3370f3a6-ccf5-41e5-9052-bb8dd0308d68-kube-api-access-2svr5\") pod \"payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7\" (UID: \"3370f3a6-ccf5-41e5-9052-bb8dd0308d68\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"
Sep 16 07:49:11.726792 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.726782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293fa14a-7b22-4b04-b747-e356dc56ee36-plugins-config-volume\") pod \"payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq\" (UID: \"293fa14a-7b22-4b04-b747-e356dc56ee36\") " pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"
Sep 16 07:49:11.726960 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.726853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3370f3a6-ccf5-41e5-9052-bb8dd0308d68-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7\" (UID: \"3370f3a6-ccf5-41e5-9052-bb8dd0308d68\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"
Sep 16 07:49:11.726960 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.726883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqt5\" (UniqueName: \"kubernetes.io/projected/293fa14a-7b22-4b04-b747-e356dc56ee36-kube-api-access-ptqt5\") pod \"payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq\" (UID: \"293fa14a-7b22-4b04-b747-e356dc56ee36\") " pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"
Sep 16 07:49:11.727657 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.727635    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293fa14a-7b22-4b04-b747-e356dc56ee36-plugins-config-volume\") pod \"payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq\" (UID: \"293fa14a-7b22-4b04-b747-e356dc56ee36\") " pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"
Sep 16 07:49:11.727691 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.727635    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3370f3a6-ccf5-41e5-9052-bb8dd0308d68-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7\" (UID: \"3370f3a6-ccf5-41e5-9052-bb8dd0308d68\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"
Sep 16 07:49:11.739754 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.739724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2svr5\" (UniqueName: \"kubernetes.io/projected/3370f3a6-ccf5-41e5-9052-bb8dd0308d68-kube-api-access-2svr5\") pod \"payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7\" (UID: \"3370f3a6-ccf5-41e5-9052-bb8dd0308d68\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"
Sep 16 07:49:11.743146 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.743128    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqt5\" (UniqueName: \"kubernetes.io/projected/293fa14a-7b22-4b04-b747-e356dc56ee36-kube-api-access-ptqt5\") pod \"payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq\" (UID: \"293fa14a-7b22-4b04-b747-e356dc56ee36\") " pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"
Sep 16 07:49:11.779534 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.779510    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"
Sep 16 07:49:11.810388 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:11.810358    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"
Sep 16 07:49:12.009641 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:12.009553    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b60434-pkhvr"
Sep 16 07:49:12.009641 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:12.009578    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b60434-pkhvr" event={"ID":"64ec391f-fd5e-490b-ad03-a3ca3d09f94f","Type":"ContainerDied","Data":"c0553ec076dcd31d5d08af3f7c021fc1bebc5758707e522057ec9ace1ded6e52"}
Sep 16 07:49:12.010272 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:12.009658    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0553ec076dcd31d5d08af3f7c021fc1bebc5758707e522057ec9ace1ded6e52"
Sep 16 07:49:12.058550 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:12.058523    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"]
Sep 16 07:49:12.060559 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:49:12.060536    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3370f3a6_ccf5_41e5_9052_bb8dd0308d68.slice/crio-037de3dd1872fe70e2f6d9232088577c82229d84bc47d3952b4a50bc9af5f641 WatchSource:0}: Error finding container 037de3dd1872fe70e2f6d9232088577c82229d84bc47d3952b4a50bc9af5f641: Status 404 returned error can't find the container with id 037de3dd1872fe70e2f6d9232088577c82229d84bc47d3952b4a50bc9af5f641
Sep 16 07:49:12.104339 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:12.104316    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"]
Sep 16 07:49:12.108892 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:49:12.108864    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293fa14a_7b22_4b04_b747_e356dc56ee36.slice/crio-ac4f9e321c1a3487065c980a0451527ef789d267bd7f84f8c067ae5fe6200f53 WatchSource:0}: Error finding container ac4f9e321c1a3487065c980a0451527ef789d267bd7f84f8c067ae5fe6200f53: Status 404 returned error can't find the container with id ac4f9e321c1a3487065c980a0451527ef789d267bd7f84f8c067ae5fe6200f53
Sep 16 07:49:13.030884 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:13.030848    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7" event={"ID":"3370f3a6-ccf5-41e5-9052-bb8dd0308d68","Type":"ContainerStarted","Data":"391d6720eb48ac539a10a681b208b879e6dca5049cb09d34c87fc0ed7f244a4e"}
Sep 16 07:49:13.031331 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:13.030936    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"
Sep 16 07:49:13.031331 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:13.030965    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7" event={"ID":"3370f3a6-ccf5-41e5-9052-bb8dd0308d68","Type":"ContainerStarted","Data":"037de3dd1872fe70e2f6d9232088577c82229d84bc47d3952b4a50bc9af5f641"}
Sep 16 07:49:13.032617 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:13.032576    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq" event={"ID":"293fa14a-7b22-4b04-b747-e356dc56ee36","Type":"ContainerStarted","Data":"d8d1d19d38006837b7d0aa2d42b3d8d71f4c5041c59c27b701f22197c05f61b3"}
Sep 16 07:49:13.032744 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:13.032628    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq" event={"ID":"293fa14a-7b22-4b04-b747-e356dc56ee36","Type":"ContainerStarted","Data":"ac4f9e321c1a3487065c980a0451527ef789d267bd7f84f8c067ae5fe6200f53"}
Sep 16 07:49:13.032744 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:13.032666    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"
Sep 16 07:49:13.047925 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:13.047878    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7" podStartSLOduration=2.047860381 podStartE2EDuration="2.047860381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:49:13.045944589 +0000 UTC m=+913.972556742" watchObservedRunningTime="2026-09-16 07:49:13.047860381 +0000 UTC m=+913.974472535"
Sep 16 07:49:13.060887 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:13.060843    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq" podStartSLOduration=2.060811129 podStartE2EDuration="2.060811129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:49:13.060757596 +0000 UTC m=+913.987369750" watchObservedRunningTime="2026-09-16 07:49:13.060811129 +0000 UTC m=+913.987423283"
Sep 16 07:49:19.082711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:19.082657    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"]
Sep 16 07:49:19.083547 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:19.083512    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g" podUID="53f89282-047a-4a97-a603-fff05b8eda4b" containerName="payload-pre-processing" containerID="cri-o://95cfff12441bf2908e8c8144a9a0cbe5545fee34a7a24f756bc7cb1579d7f508" gracePeriod=30
Sep 16 07:49:19.232499 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:19.232422    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b60434-pkhvr"]
Sep 16 07:49:19.245847 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:19.244761    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b60434-pkhvr"]
Sep 16 07:49:19.385289 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:19.385268    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"
Sep 16 07:49:19.396806 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:19.396784    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f89282-047a-4a97-a603-fff05b8eda4b-plugins-config-volume\") pod \"53f89282-047a-4a97-a603-fff05b8eda4b\" (UID: \"53f89282-047a-4a97-a603-fff05b8eda4b\") "
Sep 16 07:49:19.396970 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:19.396951    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2jrv\" (UniqueName: \"kubernetes.io/projected/53f89282-047a-4a97-a603-fff05b8eda4b-kube-api-access-q2jrv\") pod \"53f89282-047a-4a97-a603-fff05b8eda4b\" (UID: \"53f89282-047a-4a97-a603-fff05b8eda4b\") "
Sep 16 07:49:19.397178 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:19.397157    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f89282-047a-4a97-a603-fff05b8eda4b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "53f89282-047a-4a97-a603-fff05b8eda4b" (UID: "53f89282-047a-4a97-a603-fff05b8eda4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:49:19.397311 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:19.397288    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f89282-047a-4a97-a603-fff05b8eda4b-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:19.400356 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:19.400321    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f89282-047a-4a97-a603-fff05b8eda4b-kube-api-access-q2jrv" (OuterVolumeSpecName: "kube-api-access-q2jrv") pod "53f89282-047a-4a97-a603-fff05b8eda4b" (UID: "53f89282-047a-4a97-a603-fff05b8eda4b"). InnerVolumeSpecName "kube-api-access-q2jrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:49:19.498270 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:19.498237    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2jrv\" (UniqueName: \"kubernetes.io/projected/53f89282-047a-4a97-a603-fff05b8eda4b-kube-api-access-q2jrv\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:19.686082 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:19.686049    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64ec391f-fd5e-490b-ad03-a3ca3d09f94f" path="/var/lib/kubelet/pods/64ec391f-fd5e-490b-ad03-a3ca3d09f94f/volumes"
Sep 16 07:49:20.091960 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:20.091858    2581 generic.go:356] "Generic (PLEG): container finished" podID="53f89282-047a-4a97-a603-fff05b8eda4b" containerID="95cfff12441bf2908e8c8144a9a0cbe5545fee34a7a24f756bc7cb1579d7f508" exitCode=0
Sep 16 07:49:20.091960 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:20.091904    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"
Sep 16 07:49:20.091960 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:20.091940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g" event={"ID":"53f89282-047a-4a97-a603-fff05b8eda4b","Type":"ContainerDied","Data":"95cfff12441bf2908e8c8144a9a0cbe5545fee34a7a24f756bc7cb1579d7f508"}
Sep 16 07:49:20.092483 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:20.091994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g" event={"ID":"53f89282-047a-4a97-a603-fff05b8eda4b","Type":"ContainerDied","Data":"ceb778a837e9c73806f8e6eb2eaae32523a2338abc8283dd20781c6198a23dfe"}
Sep 16 07:49:20.092483 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:20.092016    2581 scope.go:117] "RemoveContainer" containerID="95cfff12441bf2908e8c8144a9a0cbe5545fee34a7a24f756bc7cb1579d7f508"
Sep 16 07:49:20.100949 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:20.100930    2581 scope.go:117] "RemoveContainer" containerID="95cfff12441bf2908e8c8144a9a0cbe5545fee34a7a24f756bc7cb1579d7f508"
Sep 16 07:49:20.101203 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:49:20.101184    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95cfff12441bf2908e8c8144a9a0cbe5545fee34a7a24f756bc7cb1579d7f508\": container with ID starting with 95cfff12441bf2908e8c8144a9a0cbe5545fee34a7a24f756bc7cb1579d7f508 not found: ID does not exist" containerID="95cfff12441bf2908e8c8144a9a0cbe5545fee34a7a24f756bc7cb1579d7f508"
Sep 16 07:49:20.101268 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:20.101214    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cfff12441bf2908e8c8144a9a0cbe5545fee34a7a24f756bc7cb1579d7f508"} err="failed to get container status \"95cfff12441bf2908e8c8144a9a0cbe5545fee34a7a24f756bc7cb1579d7f508\": rpc error: code = NotFound desc = could not find container \"95cfff12441bf2908e8c8144a9a0cbe5545fee34a7a24f756bc7cb1579d7f508\": container with ID starting with 95cfff12441bf2908e8c8144a9a0cbe5545fee34a7a24f756bc7cb1579d7f508 not found: ID does not exist"
Sep 16 07:49:20.108492 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:20.108466    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"]
Sep 16 07:49:20.113526 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:20.113500    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b60434-7657cf6bd6-cg78g"]
Sep 16 07:49:21.684010 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:21.683976    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53f89282-047a-4a97-a603-fff05b8eda4b" path="/var/lib/kubelet/pods/53f89282-047a-4a97-a603-fff05b8eda4b/volumes"
Sep 16 07:49:24.041018 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.040985    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"
Sep 16 07:49:24.041372 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.041053    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"
Sep 16 07:49:24.060840 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.060784    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk"]
Sep 16 07:49:24.061191 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.061179    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53f89282-047a-4a97-a603-fff05b8eda4b" containerName="payload-pre-processing"
Sep 16 07:49:24.061244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.061193    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="53f89282-047a-4a97-a603-fff05b8eda4b" containerName="payload-pre-processing"
Sep 16 07:49:24.061277 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.061268    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="53f89282-047a-4a97-a603-fff05b8eda4b" containerName="payload-pre-processing"
Sep 16 07:49:24.065587 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.065570    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk"
Sep 16 07:49:24.069519 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.069497    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 07:49:24.069637 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.069497    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\""
Sep 16 07:49:24.075313 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.075291    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk"]
Sep 16 07:49:24.151333 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.151293    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0a2f86-00a5-4f45-b23e-95e688dc1abe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1892c35b-xhklk\" (UID: \"db0a2f86-00a5-4f45-b23e-95e688dc1abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk"
Sep 16 07:49:24.251811 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.251777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0a2f86-00a5-4f45-b23e-95e688dc1abe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1892c35b-xhklk\" (UID: \"db0a2f86-00a5-4f45-b23e-95e688dc1abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk"
Sep 16 07:49:24.252368 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.252350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0a2f86-00a5-4f45-b23e-95e688dc1abe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1892c35b-xhklk\" (UID: \"db0a2f86-00a5-4f45-b23e-95e688dc1abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk"
Sep 16 07:49:24.377086 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.377007    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk"
Sep 16 07:49:24.570771 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.570734    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"]
Sep 16 07:49:24.669647 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:24.669619    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk"]
Sep 16 07:49:24.671295 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:49:24.671270    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0a2f86_00a5_4f45_b23e_95e688dc1abe.slice/crio-720e0ce125c92b599ef4a1edb7217c75431295c726e73d5bb3164d12b40594ac WatchSource:0}: Error finding container 720e0ce125c92b599ef4a1edb7217c75431295c726e73d5bb3164d12b40594ac: Status 404 returned error can't find the container with id 720e0ce125c92b599ef4a1edb7217c75431295c726e73d5bb3164d12b40594ac
Sep 16 07:49:25.152031 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:25.151996    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk" event={"ID":"db0a2f86-00a5-4f45-b23e-95e688dc1abe","Type":"ContainerStarted","Data":"b9b527fc10d2ba274438227cef48c23b5f39f21f1feeb88df89ff8bdcd76a963"}
Sep 16 07:49:25.152396 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:25.152069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk" event={"ID":"db0a2f86-00a5-4f45-b23e-95e688dc1abe","Type":"ContainerStarted","Data":"720e0ce125c92b599ef4a1edb7217c75431295c726e73d5bb3164d12b40594ac"}
Sep 16 07:49:25.169321 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:25.169269    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk" podStartSLOduration=1.169253562 podStartE2EDuration="1.169253562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:49:25.167315964 +0000 UTC m=+926.093928118" watchObservedRunningTime="2026-09-16 07:49:25.169253562 +0000 UTC m=+926.095865742"
Sep 16 07:49:26.160651 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:26.160614    2581 generic.go:356] "Generic (PLEG): container finished" podID="db0a2f86-00a5-4f45-b23e-95e688dc1abe" containerID="b9b527fc10d2ba274438227cef48c23b5f39f21f1feeb88df89ff8bdcd76a963" exitCode=0
Sep 16 07:49:26.161034 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:26.160683    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk" event={"ID":"db0a2f86-00a5-4f45-b23e-95e688dc1abe","Type":"ContainerDied","Data":"b9b527fc10d2ba274438227cef48c23b5f39f21f1feeb88df89ff8bdcd76a963"}
Sep 16 07:49:27.302704 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:27.302680    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk"
Sep 16 07:49:27.383390 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:27.383351    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0a2f86-00a5-4f45-b23e-95e688dc1abe-maas-api-service-ca\") pod \"db0a2f86-00a5-4f45-b23e-95e688dc1abe\" (UID: \"db0a2f86-00a5-4f45-b23e-95e688dc1abe\") "
Sep 16 07:49:27.383730 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:27.383705    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0a2f86-00a5-4f45-b23e-95e688dc1abe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "db0a2f86-00a5-4f45-b23e-95e688dc1abe" (UID: "db0a2f86-00a5-4f45-b23e-95e688dc1abe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:49:27.484407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:27.484323    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0a2f86-00a5-4f45-b23e-95e688dc1abe-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:28.172802 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:28.172766    2581 generic.go:356] "Generic (PLEG): container finished" podID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" containerID="81e19f4b371bd3859dee57dde4b0e3f3fc8b08d1e54144799f63a0aa51a77750" exitCode=0
Sep 16 07:49:28.173046 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:28.172851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk" event={"ID":"2d21ce7e-d3a1-4286-ab95-ed813cd66043","Type":"ContainerDied","Data":"81e19f4b371bd3859dee57dde4b0e3f3fc8b08d1e54144799f63a0aa51a77750"}
Sep 16 07:49:28.173553 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:28.173430    2581 scope.go:117] "RemoveContainer" containerID="81e19f4b371bd3859dee57dde4b0e3f3fc8b08d1e54144799f63a0aa51a77750"
Sep 16 07:49:28.174536 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:28.174515    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk" event={"ID":"db0a2f86-00a5-4f45-b23e-95e688dc1abe","Type":"ContainerDied","Data":"720e0ce125c92b599ef4a1edb7217c75431295c726e73d5bb3164d12b40594ac"}
Sep 16 07:49:28.174625 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:28.174539    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk"
Sep 16 07:49:28.174625 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:28.174559    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="720e0ce125c92b599ef4a1edb7217c75431295c726e73d5bb3164d12b40594ac"
Sep 16 07:49:28.174793 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:28.174757    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:49:28.174860 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:28.174847    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:49:28.174913 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:28.174881    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:49:29.181764 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:29.181733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk" event={"ID":"2d21ce7e-d3a1-4286-ab95-ed813cd66043","Type":"ContainerStarted","Data":"286d7df24f0bc47ff332ad70b6475dc5f09224c057fdc2ef3f8a07cb7e1df98d"}
Sep 16 07:49:29.657230 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:29.657195    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:29.657406 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:29.657345    2581 patch_prober.go:28] interesting pod/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 07:49:29.657406 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:29.657392    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk" podUID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 07:49:30.658110 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:30.658070    2581 patch_prober.go:28] interesting pod/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 07:49:30.658519 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:30.658128    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk" podUID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 07:49:31.658145 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:31.658106    2581 patch_prober.go:28] interesting pod/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 07:49:31.658529 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:31.658179    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk" podUID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 07:49:32.658018 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:32.657985    2581 patch_prober.go:28] interesting pod/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 07:49:32.658192 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:32.658036    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk" podUID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 07:49:33.657558 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:33.657516    2581 patch_prober.go:28] interesting pod/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 07:49:33.657731 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:33.657598    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk" podUID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 07:49:34.577627 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:34.577594    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"]
Sep 16 07:49:34.578015 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:34.577795    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq" podUID="293fa14a-7b22-4b04-b747-e356dc56ee36" containerName="payload-processing" containerID="cri-o://d8d1d19d38006837b7d0aa2d42b3d8d71f4c5041c59c27b701f22197c05f61b3" gracePeriod=30
Sep 16 07:49:34.588327 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:34.588299    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"]
Sep 16 07:49:34.588569 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:34.588545    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7" podUID="3370f3a6-ccf5-41e5-9052-bb8dd0308d68" containerName="payload-pre-processing" containerID="cri-o://391d6720eb48ac539a10a681b208b879e6dca5049cb09d34c87fc0ed7f244a4e" gracePeriod=30
Sep 16 07:49:34.657782 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:34.657717    2581 patch_prober.go:28] interesting pod/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 07:49:34.658258 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:34.658182    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk" podUID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 07:49:34.746163 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:34.746039    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk"]
Sep 16 07:49:34.748216 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:34.748189    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1892c35b-xhklk"]
Sep 16 07:49:34.889004 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:34.888963    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"]
Sep 16 07:49:34.889221 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:34.889192    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk" podUID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" containerName="istio-proxy" containerID="cri-o://286d7df24f0bc47ff332ad70b6475dc5f09224c057fdc2ef3f8a07cb7e1df98d" gracePeriod=30
Sep 16 07:49:34.955475 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:34.955427    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"
Sep 16 07:49:34.987955 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:34.986907    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"
Sep 16 07:49:35.062539 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.062508    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3370f3a6-ccf5-41e5-9052-bb8dd0308d68-plugins-config-volume\") pod \"3370f3a6-ccf5-41e5-9052-bb8dd0308d68\" (UID: \"3370f3a6-ccf5-41e5-9052-bb8dd0308d68\") "
Sep 16 07:49:35.062727 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.062554    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2svr5\" (UniqueName: \"kubernetes.io/projected/3370f3a6-ccf5-41e5-9052-bb8dd0308d68-kube-api-access-2svr5\") pod \"3370f3a6-ccf5-41e5-9052-bb8dd0308d68\" (UID: \"3370f3a6-ccf5-41e5-9052-bb8dd0308d68\") "
Sep 16 07:49:35.062727 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.062635    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293fa14a-7b22-4b04-b747-e356dc56ee36-plugins-config-volume\") pod \"293fa14a-7b22-4b04-b747-e356dc56ee36\" (UID: \"293fa14a-7b22-4b04-b747-e356dc56ee36\") "
Sep 16 07:49:35.062727 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.062653    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptqt5\" (UniqueName: \"kubernetes.io/projected/293fa14a-7b22-4b04-b747-e356dc56ee36-kube-api-access-ptqt5\") pod \"293fa14a-7b22-4b04-b747-e356dc56ee36\" (UID: \"293fa14a-7b22-4b04-b747-e356dc56ee36\") "
Sep 16 07:49:35.063042 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.063013    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293fa14a-7b22-4b04-b747-e356dc56ee36-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "293fa14a-7b22-4b04-b747-e356dc56ee36" (UID: "293fa14a-7b22-4b04-b747-e356dc56ee36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:49:35.063200 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.063180    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3370f3a6-ccf5-41e5-9052-bb8dd0308d68-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3370f3a6-ccf5-41e5-9052-bb8dd0308d68" (UID: "3370f3a6-ccf5-41e5-9052-bb8dd0308d68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:49:35.064780 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.064759    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3370f3a6-ccf5-41e5-9052-bb8dd0308d68-kube-api-access-2svr5" (OuterVolumeSpecName: "kube-api-access-2svr5") pod "3370f3a6-ccf5-41e5-9052-bb8dd0308d68" (UID: "3370f3a6-ccf5-41e5-9052-bb8dd0308d68"). InnerVolumeSpecName "kube-api-access-2svr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:49:35.065011 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.064995    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293fa14a-7b22-4b04-b747-e356dc56ee36-kube-api-access-ptqt5" (OuterVolumeSpecName: "kube-api-access-ptqt5") pod "293fa14a-7b22-4b04-b747-e356dc56ee36" (UID: "293fa14a-7b22-4b04-b747-e356dc56ee36"). InnerVolumeSpecName "kube-api-access-ptqt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:49:35.163866 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.163832    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/293fa14a-7b22-4b04-b747-e356dc56ee36-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:35.163866 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.163864    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptqt5\" (UniqueName: \"kubernetes.io/projected/293fa14a-7b22-4b04-b747-e356dc56ee36-kube-api-access-ptqt5\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:35.164080 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.163875    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3370f3a6-ccf5-41e5-9052-bb8dd0308d68-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:35.164080 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.163884    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2svr5\" (UniqueName: \"kubernetes.io/projected/3370f3a6-ccf5-41e5-9052-bb8dd0308d68-kube-api-access-2svr5\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:35.239844 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.236559    2581 generic.go:356] "Generic (PLEG): container finished" podID="3370f3a6-ccf5-41e5-9052-bb8dd0308d68" containerID="391d6720eb48ac539a10a681b208b879e6dca5049cb09d34c87fc0ed7f244a4e" exitCode=0
Sep 16 07:49:35.239844 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.236671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7" event={"ID":"3370f3a6-ccf5-41e5-9052-bb8dd0308d68","Type":"ContainerDied","Data":"391d6720eb48ac539a10a681b208b879e6dca5049cb09d34c87fc0ed7f244a4e"}
Sep 16 07:49:35.239844 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.236722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7" event={"ID":"3370f3a6-ccf5-41e5-9052-bb8dd0308d68","Type":"ContainerDied","Data":"037de3dd1872fe70e2f6d9232088577c82229d84bc47d3952b4a50bc9af5f641"}
Sep 16 07:49:35.239844 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.236756    2581 scope.go:117] "RemoveContainer" containerID="391d6720eb48ac539a10a681b208b879e6dca5049cb09d34c87fc0ed7f244a4e"
Sep 16 07:49:35.239844 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.236958    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"
Sep 16 07:49:35.252450 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.252417    2581 generic.go:356] "Generic (PLEG): container finished" podID="293fa14a-7b22-4b04-b747-e356dc56ee36" containerID="d8d1d19d38006837b7d0aa2d42b3d8d71f4c5041c59c27b701f22197c05f61b3" exitCode=0
Sep 16 07:49:35.252680 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.252659    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq" event={"ID":"293fa14a-7b22-4b04-b747-e356dc56ee36","Type":"ContainerDied","Data":"d8d1d19d38006837b7d0aa2d42b3d8d71f4c5041c59c27b701f22197c05f61b3"}
Sep 16 07:49:35.252875 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.252811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq" event={"ID":"293fa14a-7b22-4b04-b747-e356dc56ee36","Type":"ContainerDied","Data":"ac4f9e321c1a3487065c980a0451527ef789d267bd7f84f8c067ae5fe6200f53"}
Sep 16 07:49:35.252975 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.252943    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"
Sep 16 07:49:35.260107 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.260085    2581 scope.go:117] "RemoveContainer" containerID="391d6720eb48ac539a10a681b208b879e6dca5049cb09d34c87fc0ed7f244a4e"
Sep 16 07:49:35.260395 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:49:35.260375    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391d6720eb48ac539a10a681b208b879e6dca5049cb09d34c87fc0ed7f244a4e\": container with ID starting with 391d6720eb48ac539a10a681b208b879e6dca5049cb09d34c87fc0ed7f244a4e not found: ID does not exist" containerID="391d6720eb48ac539a10a681b208b879e6dca5049cb09d34c87fc0ed7f244a4e"
Sep 16 07:49:35.260469 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.260403    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391d6720eb48ac539a10a681b208b879e6dca5049cb09d34c87fc0ed7f244a4e"} err="failed to get container status \"391d6720eb48ac539a10a681b208b879e6dca5049cb09d34c87fc0ed7f244a4e\": rpc error: code = NotFound desc = could not find container \"391d6720eb48ac539a10a681b208b879e6dca5049cb09d34c87fc0ed7f244a4e\": container with ID starting with 391d6720eb48ac539a10a681b208b879e6dca5049cb09d34c87fc0ed7f244a4e not found: ID does not exist"
Sep 16 07:49:35.260469 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.260420    2581 scope.go:117] "RemoveContainer" containerID="d8d1d19d38006837b7d0aa2d42b3d8d71f4c5041c59c27b701f22197c05f61b3"
Sep 16 07:49:35.268901 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.268881    2581 scope.go:117] "RemoveContainer" containerID="d8d1d19d38006837b7d0aa2d42b3d8d71f4c5041c59c27b701f22197c05f61b3"
Sep 16 07:49:35.269173 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:49:35.269151    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d1d19d38006837b7d0aa2d42b3d8d71f4c5041c59c27b701f22197c05f61b3\": container with ID starting with d8d1d19d38006837b7d0aa2d42b3d8d71f4c5041c59c27b701f22197c05f61b3 not found: ID does not exist" containerID="d8d1d19d38006837b7d0aa2d42b3d8d71f4c5041c59c27b701f22197c05f61b3"
Sep 16 07:49:35.269215 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.269181    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d1d19d38006837b7d0aa2d42b3d8d71f4c5041c59c27b701f22197c05f61b3"} err="failed to get container status \"d8d1d19d38006837b7d0aa2d42b3d8d71f4c5041c59c27b701f22197c05f61b3\": rpc error: code = NotFound desc = could not find container \"d8d1d19d38006837b7d0aa2d42b3d8d71f4c5041c59c27b701f22197c05f61b3\": container with ID starting with d8d1d19d38006837b7d0aa2d42b3d8d71f4c5041c59c27b701f22197c05f61b3 not found: ID does not exist"
Sep 16 07:49:35.272857 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.272809    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"]
Sep 16 07:49:35.276370 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.276349    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1892c35b-5b48475b66-5lgv7"]
Sep 16 07:49:35.286256 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.286232    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"]
Sep 16 07:49:35.292467 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.292416    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1892c35b-5c64874b89-65zbq"]
Sep 16 07:49:35.685134 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.685103    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="293fa14a-7b22-4b04-b747-e356dc56ee36" path="/var/lib/kubelet/pods/293fa14a-7b22-4b04-b747-e356dc56ee36/volumes"
Sep 16 07:49:35.685490 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.685452    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3370f3a6-ccf5-41e5-9052-bb8dd0308d68" path="/var/lib/kubelet/pods/3370f3a6-ccf5-41e5-9052-bb8dd0308d68/volumes"
Sep 16 07:49:35.685771 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:35.685757    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db0a2f86-00a5-4f45-b23e-95e688dc1abe" path="/var/lib/kubelet/pods/db0a2f86-00a5-4f45-b23e-95e688dc1abe/volumes"
Sep 16 07:49:38.989557 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:38.989524    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:39.590434 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.590398    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx" podUID="fb2f87c8-421c-4f0c-b0ae-8ebfad69370c" containerName="main" containerID="cri-o://65d7d3fb15163acb8839398094dd5cf228679383c6d0daedd22304dbcc813a79" gracePeriod=45
Sep 16 07:49:39.841577 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.841523    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:49:39.911001 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.910971    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-tmp-dir\") pod \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") "
Sep 16 07:49:39.911162 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.911055    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwmh6\" (UniqueName: \"kubernetes.io/projected/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-kube-api-access-kwmh6\") pod \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") "
Sep 16 07:49:39.911162 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.911109    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-model-cache\") pod \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") "
Sep 16 07:49:39.911295 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.911161    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-home\") pod \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") "
Sep 16 07:49:39.911295 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.911203    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-tls-certs\") pod \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") "
Sep 16 07:49:39.911295 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.911230    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-dshm\") pod \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\" (UID: \"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c\") "
Sep 16 07:49:39.911454 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.911323    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb2f87c8-421c-4f0c-b0ae-8ebfad69370c" (UID: "fb2f87c8-421c-4f0c-b0ae-8ebfad69370c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:49:39.911454 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.911362    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb2f87c8-421c-4f0c-b0ae-8ebfad69370c" (UID: "fb2f87c8-421c-4f0c-b0ae-8ebfad69370c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:49:39.911564 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.911504    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-home" (OuterVolumeSpecName: "home") pod "fb2f87c8-421c-4f0c-b0ae-8ebfad69370c" (UID: "fb2f87c8-421c-4f0c-b0ae-8ebfad69370c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:49:39.911796 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.911591    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:39.911796 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.911609    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:39.911796 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.911620    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:39.913574 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.913542    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb2f87c8-421c-4f0c-b0ae-8ebfad69370c" (UID: "fb2f87c8-421c-4f0c-b0ae-8ebfad69370c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 07:49:39.913764 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.913743    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-kube-api-access-kwmh6" (OuterVolumeSpecName: "kube-api-access-kwmh6") pod "fb2f87c8-421c-4f0c-b0ae-8ebfad69370c" (UID: "fb2f87c8-421c-4f0c-b0ae-8ebfad69370c"). InnerVolumeSpecName "kube-api-access-kwmh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:49:39.913909 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:39.913885    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-dshm" (OuterVolumeSpecName: "dshm") pod "fb2f87c8-421c-4f0c-b0ae-8ebfad69370c" (UID: "fb2f87c8-421c-4f0c-b0ae-8ebfad69370c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:49:40.012998 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.012967    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:40.012998 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.012992    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:40.012998 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.013002    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwmh6\" (UniqueName: \"kubernetes.io/projected/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c-kube-api-access-kwmh6\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:40.280961 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.280923    2581 generic.go:356] "Generic (PLEG): container finished" podID="fb2f87c8-421c-4f0c-b0ae-8ebfad69370c" containerID="65d7d3fb15163acb8839398094dd5cf228679383c6d0daedd22304dbcc813a79" exitCode=0
Sep 16 07:49:40.280961 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.280958    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"
Sep 16 07:49:40.281172 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.280976    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx" event={"ID":"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c","Type":"ContainerDied","Data":"65d7d3fb15163acb8839398094dd5cf228679383c6d0daedd22304dbcc813a79"}
Sep 16 07:49:40.281172 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.281012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx" event={"ID":"fb2f87c8-421c-4f0c-b0ae-8ebfad69370c","Type":"ContainerDied","Data":"5174504bc934a7ab267cd92992f46c9526206ad26bebd948e8bfa5b549122d34"}
Sep 16 07:49:40.281172 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.281029    2581 scope.go:117] "RemoveContainer" containerID="65d7d3fb15163acb8839398094dd5cf228679383c6d0daedd22304dbcc813a79"
Sep 16 07:49:40.290215 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.290197    2581 scope.go:117] "RemoveContainer" containerID="65d7d3fb15163acb8839398094dd5cf228679383c6d0daedd22304dbcc813a79"
Sep 16 07:49:40.290487 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:49:40.290470    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d7d3fb15163acb8839398094dd5cf228679383c6d0daedd22304dbcc813a79\": container with ID starting with 65d7d3fb15163acb8839398094dd5cf228679383c6d0daedd22304dbcc813a79 not found: ID does not exist" containerID="65d7d3fb15163acb8839398094dd5cf228679383c6d0daedd22304dbcc813a79"
Sep 16 07:49:40.290533 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.290494    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d7d3fb15163acb8839398094dd5cf228679383c6d0daedd22304dbcc813a79"} err="failed to get container status \"65d7d3fb15163acb8839398094dd5cf228679383c6d0daedd22304dbcc813a79\": rpc error: code = NotFound desc = could not find container \"65d7d3fb15163acb8839398094dd5cf228679383c6d0daedd22304dbcc813a79\": container with ID starting with 65d7d3fb15163acb8839398094dd5cf228679383c6d0daedd22304dbcc813a79 not found: ID does not exist"
Sep 16 07:49:40.301345 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.301320    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"]
Sep 16 07:49:40.305256 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.305229    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-b60434/autores-model-b60434-kserve-7bb65b97f5-kvhxx"]
Sep 16 07:49:40.728784 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.728753    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:40.820975 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.820945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-token\") pod \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") "
Sep 16 07:49:40.821167 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821032    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-data\") pod \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") "
Sep 16 07:49:40.821167 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821051    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-envoy\") pod \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") "
Sep 16 07:49:40.821167 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821082    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-workload-certs\") pod \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") "
Sep 16 07:49:40.821167 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821111    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-workload-socket\") pod \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") "
Sep 16 07:49:40.821167 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821137    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-credential-socket\") pod \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") "
Sep 16 07:49:40.821167 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821163    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istiod-ca-cert\") pod \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") "
Sep 16 07:49:40.821497 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821191    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj57f\" (UniqueName: \"kubernetes.io/projected/2d21ce7e-d3a1-4286-ab95-ed813cd66043-kube-api-access-cj57f\") pod \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") "
Sep 16 07:49:40.821497 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821216    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-podinfo\") pod \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\" (UID: \"2d21ce7e-d3a1-4286-ab95-ed813cd66043\") "
Sep 16 07:49:40.821497 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821419    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d21ce7e-d3a1-4286-ab95-ed813cd66043" (UID: "2d21ce7e-d3a1-4286-ab95-ed813cd66043"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:49:40.821497 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821427    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d21ce7e-d3a1-4286-ab95-ed813cd66043" (UID: "2d21ce7e-d3a1-4286-ab95-ed813cd66043"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:49:40.821497 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d21ce7e-d3a1-4286-ab95-ed813cd66043" (UID: "2d21ce7e-d3a1-4286-ab95-ed813cd66043"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:49:40.821748 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821546    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d21ce7e-d3a1-4286-ab95-ed813cd66043" (UID: "2d21ce7e-d3a1-4286-ab95-ed813cd66043"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:49:40.821748 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821554    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:40.821748 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821602    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:40.821748 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821619    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:40.821748 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.821628    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d21ce7e-d3a1-4286-ab95-ed813cd66043" (UID: "2d21ce7e-d3a1-4286-ab95-ed813cd66043"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:49:40.823306 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.823280    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d21ce7e-d3a1-4286-ab95-ed813cd66043" (UID: "2d21ce7e-d3a1-4286-ab95-ed813cd66043"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:49:40.823439 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.823415    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d21ce7e-d3a1-4286-ab95-ed813cd66043" (UID: "2d21ce7e-d3a1-4286-ab95-ed813cd66043"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:49:40.823513 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.823497    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d21ce7e-d3a1-4286-ab95-ed813cd66043-kube-api-access-cj57f" (OuterVolumeSpecName: "kube-api-access-cj57f") pod "2d21ce7e-d3a1-4286-ab95-ed813cd66043" (UID: "2d21ce7e-d3a1-4286-ab95-ed813cd66043"). InnerVolumeSpecName "kube-api-access-cj57f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:49:40.823593 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.823575    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d21ce7e-d3a1-4286-ab95-ed813cd66043" (UID: "2d21ce7e-d3a1-4286-ab95-ed813cd66043"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 07:49:40.922858 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.922806    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:40.922858 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.922858    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d21ce7e-d3a1-4286-ab95-ed813cd66043-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:40.923075 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.922869    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:40.923075 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.922878    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj57f\" (UniqueName: \"kubernetes.io/projected/2d21ce7e-d3a1-4286-ab95-ed813cd66043-kube-api-access-cj57f\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:40.923075 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.922888    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:40.923075 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:40.922896    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d21ce7e-d3a1-4286-ab95-ed813cd66043-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:49:41.289088 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:41.289000    2581 generic.go:356] "Generic (PLEG): container finished" podID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" containerID="286d7df24f0bc47ff332ad70b6475dc5f09224c057fdc2ef3f8a07cb7e1df98d" exitCode=0
Sep 16 07:49:41.289088 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:41.289075    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"
Sep 16 07:49:41.289549 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:41.289084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk" event={"ID":"2d21ce7e-d3a1-4286-ab95-ed813cd66043","Type":"ContainerDied","Data":"286d7df24f0bc47ff332ad70b6475dc5f09224c057fdc2ef3f8a07cb7e1df98d"}
Sep 16 07:49:41.289549 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:41.289168    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk" event={"ID":"2d21ce7e-d3a1-4286-ab95-ed813cd66043","Type":"ContainerDied","Data":"cc72e586a0556d7ff00614072fec56a5196f1396ef5c44ff8b424d44b2c6fc2c"}
Sep 16 07:49:41.289549 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:41.289192    2581 scope.go:117] "RemoveContainer" containerID="286d7df24f0bc47ff332ad70b6475dc5f09224c057fdc2ef3f8a07cb7e1df98d"
Sep 16 07:49:41.298669 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:41.298652    2581 scope.go:117] "RemoveContainer" containerID="81e19f4b371bd3859dee57dde4b0e3f3fc8b08d1e54144799f63a0aa51a77750"
Sep 16 07:49:41.306583 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:41.306564    2581 scope.go:117] "RemoveContainer" containerID="286d7df24f0bc47ff332ad70b6475dc5f09224c057fdc2ef3f8a07cb7e1df98d"
Sep 16 07:49:41.306880 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:49:41.306856    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286d7df24f0bc47ff332ad70b6475dc5f09224c057fdc2ef3f8a07cb7e1df98d\": container with ID starting with 286d7df24f0bc47ff332ad70b6475dc5f09224c057fdc2ef3f8a07cb7e1df98d not found: ID does not exist" containerID="286d7df24f0bc47ff332ad70b6475dc5f09224c057fdc2ef3f8a07cb7e1df98d"
Sep 16 07:49:41.306996 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:41.306886    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286d7df24f0bc47ff332ad70b6475dc5f09224c057fdc2ef3f8a07cb7e1df98d"} err="failed to get container status \"286d7df24f0bc47ff332ad70b6475dc5f09224c057fdc2ef3f8a07cb7e1df98d\": rpc error: code = NotFound desc = could not find container \"286d7df24f0bc47ff332ad70b6475dc5f09224c057fdc2ef3f8a07cb7e1df98d\": container with ID starting with 286d7df24f0bc47ff332ad70b6475dc5f09224c057fdc2ef3f8a07cb7e1df98d not found: ID does not exist"
Sep 16 07:49:41.306996 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:41.306903    2581 scope.go:117] "RemoveContainer" containerID="81e19f4b371bd3859dee57dde4b0e3f3fc8b08d1e54144799f63a0aa51a77750"
Sep 16 07:49:41.307150 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:49:41.307130    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e19f4b371bd3859dee57dde4b0e3f3fc8b08d1e54144799f63a0aa51a77750\": container with ID starting with 81e19f4b371bd3859dee57dde4b0e3f3fc8b08d1e54144799f63a0aa51a77750 not found: ID does not exist" containerID="81e19f4b371bd3859dee57dde4b0e3f3fc8b08d1e54144799f63a0aa51a77750"
Sep 16 07:49:41.307203 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:41.307155    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e19f4b371bd3859dee57dde4b0e3f3fc8b08d1e54144799f63a0aa51a77750"} err="failed to get container status \"81e19f4b371bd3859dee57dde4b0e3f3fc8b08d1e54144799f63a0aa51a77750\": rpc error: code = NotFound desc = could not find container \"81e19f4b371bd3859dee57dde4b0e3f3fc8b08d1e54144799f63a0aa51a77750\": container with ID starting with 81e19f4b371bd3859dee57dde4b0e3f3fc8b08d1e54144799f63a0aa51a77750 not found: ID does not exist"
Sep 16 07:49:41.310981 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:41.310959    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"]
Sep 16 07:49:41.316613 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:41.316593    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1892c35b-openshift-default-689c8ffd7b-6mcfk"]
Sep 16 07:49:41.684198 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:41.684171    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" path="/var/lib/kubelet/pods/2d21ce7e-d3a1-4286-ab95-ed813cd66043/volumes"
Sep 16 07:49:41.684579 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:49:41.684567    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb2f87c8-421c-4f0c-b0ae-8ebfad69370c" path="/var/lib/kubelet/pods/fb2f87c8-421c-4f0c-b0ae-8ebfad69370c/volumes"
Sep 16 07:50:02.580043 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580002    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"]
Sep 16 07:50:02.580579 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580564    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" containerName="istio-proxy"
Sep 16 07:50:02.580624 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580583    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" containerName="istio-proxy"
Sep 16 07:50:02.580624 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580598    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" containerName="istio-proxy"
Sep 16 07:50:02.580624 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580607    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" containerName="istio-proxy"
Sep 16 07:50:02.580624 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580620    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3370f3a6-ccf5-41e5-9052-bb8dd0308d68" containerName="payload-pre-processing"
Sep 16 07:50:02.580749 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580629    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3370f3a6-ccf5-41e5-9052-bb8dd0308d68" containerName="payload-pre-processing"
Sep 16 07:50:02.580749 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580639    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="293fa14a-7b22-4b04-b747-e356dc56ee36" containerName="payload-processing"
Sep 16 07:50:02.580749 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580647    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="293fa14a-7b22-4b04-b747-e356dc56ee36" containerName="payload-processing"
Sep 16 07:50:02.580749 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580658    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db0a2f86-00a5-4f45-b23e-95e688dc1abe" containerName="revoke-keys"
Sep 16 07:50:02.580749 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580666    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="db0a2f86-00a5-4f45-b23e-95e688dc1abe" containerName="revoke-keys"
Sep 16 07:50:02.580749 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580682    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb2f87c8-421c-4f0c-b0ae-8ebfad69370c" containerName="main"
Sep 16 07:50:02.580749 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580690    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb2f87c8-421c-4f0c-b0ae-8ebfad69370c" containerName="main"
Sep 16 07:50:02.580988 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580789    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb2f87c8-421c-4f0c-b0ae-8ebfad69370c" containerName="main"
Sep 16 07:50:02.580988 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580801    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3370f3a6-ccf5-41e5-9052-bb8dd0308d68" containerName="payload-pre-processing"
Sep 16 07:50:02.580988 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580809    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" containerName="istio-proxy"
Sep 16 07:50:02.580988 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580833    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="db0a2f86-00a5-4f45-b23e-95e688dc1abe" containerName="revoke-keys"
Sep 16 07:50:02.580988 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580844    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="293fa14a-7b22-4b04-b747-e356dc56ee36" containerName="payload-processing"
Sep 16 07:50:02.580988 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.580853    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d21ce7e-d3a1-4286-ab95-ed813cd66043" containerName="istio-proxy"
Sep 16 07:50:02.584183 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.584164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.587123 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.587059    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-82bbbf-openshift-default-dockercfg-vc8jz\""
Sep 16 07:50:02.596018 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.595994    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"]
Sep 16 07:50:02.733102 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.733067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4509ac-8880-4728-8620-85e445069160-istiod-ca-cert\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.733253 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.733138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4509ac-8880-4728-8620-85e445069160-istio-token\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.733253 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.733181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqlqz\" (UniqueName: \"kubernetes.io/projected/0c4509ac-8880-4728-8620-85e445069160-kube-api-access-wqlqz\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.733253 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.733211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-istio-data\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.733374 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.733285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4509ac-8880-4728-8620-85e445069160-istio-podinfo\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.733374 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.733327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-workload-socket\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.733374 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.733344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-credential-socket\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.733374 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.733364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-istio-envoy\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.733497 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.733394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-workload-certs\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.834503 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.834418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4509ac-8880-4728-8620-85e445069160-istiod-ca-cert\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.834503 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.834478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4509ac-8880-4728-8620-85e445069160-istio-token\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.834739 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.834504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqlqz\" (UniqueName: \"kubernetes.io/projected/0c4509ac-8880-4728-8620-85e445069160-kube-api-access-wqlqz\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.834739 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.834521    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-istio-data\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.834739 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.834549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4509ac-8880-4728-8620-85e445069160-istio-podinfo\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.834739 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.834576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-workload-socket\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.834739 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.834626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-credential-socket\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.834739 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.834671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-istio-envoy\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.834739 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.834720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-workload-certs\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.835125 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.835028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-istio-data\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.835125 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.835096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-workload-certs\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.835125 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.835109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-credential-socket\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.835229 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.835145    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-workload-socket\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.835285 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.835266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4509ac-8880-4728-8620-85e445069160-istiod-ca-cert\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.837170 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.837146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-istio-envoy\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.837364 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.837327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4509ac-8880-4728-8620-85e445069160-istio-podinfo\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.844101 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.844080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4509ac-8880-4728-8620-85e445069160-istio-token\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.844522 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.844502    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqlqz\" (UniqueName: \"kubernetes.io/projected/0c4509ac-8880-4728-8620-85e445069160-kube-api-access-wqlqz\") pod \"e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") " pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:02.898682 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:02.898655    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:03.179866 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:03.179839    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"]
Sep 16 07:50:03.181575 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:50:03.181551    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4509ac_8880_4728_8620_85e445069160.slice/crio-4ab0719481b394c4f3592c4cae906ed073f84f37b64d40a62b846332617e1b2f WatchSource:0}: Error finding container 4ab0719481b394c4f3592c4cae906ed073f84f37b64d40a62b846332617e1b2f: Status 404 returned error can't find the container with id 4ab0719481b394c4f3592c4cae906ed073f84f37b64d40a62b846332617e1b2f
Sep 16 07:50:03.184260 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:03.184226    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:50:03.184369 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:03.184288    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:50:03.184369 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:03.184313    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:50:03.403844 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:03.403786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2" event={"ID":"0c4509ac-8880-4728-8620-85e445069160","Type":"ContainerStarted","Data":"d8ff7e7327a7fd5e3d326d3258ce0183496d59bc72c0950def2af32ed25e5f6e"}
Sep 16 07:50:03.404015 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:03.403858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2" event={"ID":"0c4509ac-8880-4728-8620-85e445069160","Type":"ContainerStarted","Data":"4ab0719481b394c4f3592c4cae906ed073f84f37b64d40a62b846332617e1b2f"}
Sep 16 07:50:03.427968 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:03.427911    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2" podStartSLOduration=1.427891325 podStartE2EDuration="1.427891325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:50:03.425214026 +0000 UTC m=+964.351826203" watchObservedRunningTime="2026-09-16 07:50:03.427891325 +0000 UTC m=+964.354503478"
Sep 16 07:50:03.899052 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:03.899020    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:04.904572 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:04.904543    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:05.415179 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:05.415147    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:05.416202 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:05.416177    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:09.129936 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.129902    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"]
Sep 16 07:50:09.132758 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.132738    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"
Sep 16 07:50:09.135276 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.135257    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-82bbbf\""
Sep 16 07:50:09.135376 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.135347    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-82bbbf-dockercfg-btvsq\""
Sep 16 07:50:09.145295 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.145273    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"]
Sep 16 07:50:09.174498 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.174469    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"]
Sep 16 07:50:09.176865 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.176849    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"
Sep 16 07:50:09.189745 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.189719    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"]
Sep 16 07:50:09.195570 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.195534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wwnl\" (UniqueName: \"kubernetes.io/projected/afe3901d-6928-46d9-9f7b-c7f4a5b0f599-kube-api-access-7wwnl\") pod \"payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6\" (UID: \"afe3901d-6928-46d9-9f7b-c7f4a5b0f599\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"
Sep 16 07:50:09.195692 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.195577    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe3901d-6928-46d9-9f7b-c7f4a5b0f599-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6\" (UID: \"afe3901d-6928-46d9-9f7b-c7f4a5b0f599\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"
Sep 16 07:50:09.195692 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.195625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dadab9e0-00ff-40c8-ad4b-788e17742890-plugins-config-volume\") pod \"payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl\" (UID: \"dadab9e0-00ff-40c8-ad4b-788e17742890\") " pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"
Sep 16 07:50:09.195802 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.195699    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxc66\" (UniqueName: \"kubernetes.io/projected/dadab9e0-00ff-40c8-ad4b-788e17742890-kube-api-access-qxc66\") pod \"payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl\" (UID: \"dadab9e0-00ff-40c8-ad4b-788e17742890\") " pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"
Sep 16 07:50:09.296494 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.296465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wwnl\" (UniqueName: \"kubernetes.io/projected/afe3901d-6928-46d9-9f7b-c7f4a5b0f599-kube-api-access-7wwnl\") pod \"payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6\" (UID: \"afe3901d-6928-46d9-9f7b-c7f4a5b0f599\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"
Sep 16 07:50:09.296690 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.296507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe3901d-6928-46d9-9f7b-c7f4a5b0f599-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6\" (UID: \"afe3901d-6928-46d9-9f7b-c7f4a5b0f599\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"
Sep 16 07:50:09.296690 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.296545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dadab9e0-00ff-40c8-ad4b-788e17742890-plugins-config-volume\") pod \"payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl\" (UID: \"dadab9e0-00ff-40c8-ad4b-788e17742890\") " pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"
Sep 16 07:50:09.296690 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.296589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxc66\" (UniqueName: \"kubernetes.io/projected/dadab9e0-00ff-40c8-ad4b-788e17742890-kube-api-access-qxc66\") pod \"payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl\" (UID: \"dadab9e0-00ff-40c8-ad4b-788e17742890\") " pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"
Sep 16 07:50:09.297379 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.297355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe3901d-6928-46d9-9f7b-c7f4a5b0f599-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6\" (UID: \"afe3901d-6928-46d9-9f7b-c7f4a5b0f599\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"
Sep 16 07:50:09.297464 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.297354    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dadab9e0-00ff-40c8-ad4b-788e17742890-plugins-config-volume\") pod \"payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl\" (UID: \"dadab9e0-00ff-40c8-ad4b-788e17742890\") " pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"
Sep 16 07:50:09.304889 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.304868    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wwnl\" (UniqueName: \"kubernetes.io/projected/afe3901d-6928-46d9-9f7b-c7f4a5b0f599-kube-api-access-7wwnl\") pod \"payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6\" (UID: \"afe3901d-6928-46d9-9f7b-c7f4a5b0f599\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"
Sep 16 07:50:09.304980 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.304952    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxc66\" (UniqueName: \"kubernetes.io/projected/dadab9e0-00ff-40c8-ad4b-788e17742890-kube-api-access-qxc66\") pod \"payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl\" (UID: \"dadab9e0-00ff-40c8-ad4b-788e17742890\") " pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"
Sep 16 07:50:09.442964 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.442934    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"
Sep 16 07:50:09.486494 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.486459    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"
Sep 16 07:50:09.714250 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.714223    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"]
Sep 16 07:50:09.715578 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:50:09.715554    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe3901d_6928_46d9_9f7b_c7f4a5b0f599.slice/crio-69b4525fb88a2834d5e7b33fbafdc7f0be51d2d2f774123ad55ee0790587aa1f WatchSource:0}: Error finding container 69b4525fb88a2834d5e7b33fbafdc7f0be51d2d2f774123ad55ee0790587aa1f: Status 404 returned error can't find the container with id 69b4525fb88a2834d5e7b33fbafdc7f0be51d2d2f774123ad55ee0790587aa1f
Sep 16 07:50:09.763930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:09.763910    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"]
Sep 16 07:50:09.770751 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:50:09.770728    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadab9e0_00ff_40c8_ad4b_788e17742890.slice/crio-ded1e7a7e26c9a382ab0c3be5f59095adc07f31256638019af2cb5fc5f5f578f WatchSource:0}: Error finding container ded1e7a7e26c9a382ab0c3be5f59095adc07f31256638019af2cb5fc5f5f578f: Status 404 returned error can't find the container with id ded1e7a7e26c9a382ab0c3be5f59095adc07f31256638019af2cb5fc5f5f578f
Sep 16 07:50:10.445648 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:10.445604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl" event={"ID":"dadab9e0-00ff-40c8-ad4b-788e17742890","Type":"ContainerStarted","Data":"587c328cd3c84336a223d70148d3928625dda54e583193a020b1cf9421ed7be7"}
Sep 16 07:50:10.446117 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:10.445678    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"
Sep 16 07:50:10.446117 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:10.445701    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl" event={"ID":"dadab9e0-00ff-40c8-ad4b-788e17742890","Type":"ContainerStarted","Data":"ded1e7a7e26c9a382ab0c3be5f59095adc07f31256638019af2cb5fc5f5f578f"}
Sep 16 07:50:10.447471 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:10.447444    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6" event={"ID":"afe3901d-6928-46d9-9f7b-c7f4a5b0f599","Type":"ContainerStarted","Data":"e058cf3a8fe9cd13caf3dc37b3dee30c74092076c14da45cb7b23f1db0588d90"}
Sep 16 07:50:10.447588 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:10.447485    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6" event={"ID":"afe3901d-6928-46d9-9f7b-c7f4a5b0f599","Type":"ContainerStarted","Data":"69b4525fb88a2834d5e7b33fbafdc7f0be51d2d2f774123ad55ee0790587aa1f"}
Sep 16 07:50:10.447588 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:10.447572    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"
Sep 16 07:50:10.461538 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:10.461499    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl" podStartSLOduration=1.46148222 podStartE2EDuration="1.46148222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:50:10.460914969 +0000 UTC m=+971.387527122" watchObservedRunningTime="2026-09-16 07:50:10.46148222 +0000 UTC m=+971.388094377"
Sep 16 07:50:10.482141 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:10.482100    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6" podStartSLOduration=1.482086032 podStartE2EDuration="1.482086032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:50:10.479354439 +0000 UTC m=+971.405966593" watchObservedRunningTime="2026-09-16 07:50:10.482086032 +0000 UTC m=+971.408698188"
Sep 16 07:50:21.455491 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:21.455463    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"
Sep 16 07:50:21.455985 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:21.455529    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"
Sep 16 07:50:28.549541 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:28.549465    2581 generic.go:356] "Generic (PLEG): container finished" podID="0c4509ac-8880-4728-8620-85e445069160" containerID="d8ff7e7327a7fd5e3d326d3258ce0183496d59bc72c0950def2af32ed25e5f6e" exitCode=0
Sep 16 07:50:28.550132 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:28.549551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2" event={"ID":"0c4509ac-8880-4728-8620-85e445069160","Type":"ContainerDied","Data":"d8ff7e7327a7fd5e3d326d3258ce0183496d59bc72c0950def2af32ed25e5f6e"}
Sep 16 07:50:28.550132 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:28.550067    2581 scope.go:117] "RemoveContainer" containerID="d8ff7e7327a7fd5e3d326d3258ce0183496d59bc72c0950def2af32ed25e5f6e"
Sep 16 07:50:28.552262 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:28.551901    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:50:28.552262 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:28.551995    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:50:28.552262 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:28.552055    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:50:29.557131 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:29.557097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2" event={"ID":"0c4509ac-8880-4728-8620-85e445069160","Type":"ContainerStarted","Data":"271e425c851a557d5b29690799322073f063920ffe576ac4e0d247a6cdc7d18c"}
Sep 16 07:50:29.898883 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:29.898776    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:29.899074 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:29.898905    2581 patch_prober.go:28] interesting pod/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 16 07:50:29.899074 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:29.898961    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2" podUID="0c4509ac-8880-4728-8620-85e445069160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 16 07:50:30.899041 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:30.899008    2581 patch_prober.go:28] interesting pod/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 16 07:50:30.899424 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:30.899064    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2" podUID="0c4509ac-8880-4728-8620-85e445069160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 16 07:50:31.899911 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:31.899879    2581 patch_prober.go:28] interesting pod/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 16 07:50:31.900288 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:31.899931    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2" podUID="0c4509ac-8880-4728-8620-85e445069160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 16 07:50:32.900002 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:32.899964    2581 patch_prober.go:28] interesting pod/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 16 07:50:32.900385 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:32.900037    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2" podUID="0c4509ac-8880-4728-8620-85e445069160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 16 07:50:33.899370 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:33.899338    2581 patch_prober.go:28] interesting pod/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 16 07:50:33.899559 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:33.899398    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2" podUID="0c4509ac-8880-4728-8620-85e445069160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 16 07:50:34.898972 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:34.898935    2581 patch_prober.go:28] interesting pod/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 16 07:50:34.899357 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:34.899000    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2" podUID="0c4509ac-8880-4728-8620-85e445069160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 16 07:50:35.415699 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:35.415662    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:35.903155 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:35.903075    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:50:35.904116 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:50:35.904097    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:51:01.967986 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:01.967952    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"]
Sep 16 07:51:01.971276 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:01.971253    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:01.973681 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:01.973661    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dd359c\"/\"kube-root-ca.crt\""
Sep 16 07:51:01.973997 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:01.973978    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dd359c\"/\"openshift-service-ca.crt\""
Sep 16 07:51:01.975013 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:01.974994    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dd359c\"/\"default-dockercfg-7vk4x\""
Sep 16 07:51:01.975110 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:01.975075    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dd359c\"/\"test-model-dd359c-kserve-self-signed-certs\""
Sep 16 07:51:01.980206 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:01.980183    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"]
Sep 16 07:51:02.126980 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.126941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-tmp-dir\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.127266 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.127242    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-model-cache\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.127424 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.127407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-dshm\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.127571 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.127556    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-home\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.127675 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.127659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8gkf\" (UniqueName: \"kubernetes.io/projected/2013af00-9176-4429-8d9c-9e977456d921-kube-api-access-n8gkf\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.127853 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.127836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2013af00-9176-4429-8d9c-9e977456d921-tls-certs\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.229105 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.229018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-dshm\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.229105 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.229070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-home\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.229105 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.229088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8gkf\" (UniqueName: \"kubernetes.io/projected/2013af00-9176-4429-8d9c-9e977456d921-kube-api-access-n8gkf\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.229345 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.229128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2013af00-9176-4429-8d9c-9e977456d921-tls-certs\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.229345 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.229170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-tmp-dir\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.229345 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.229203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-model-cache\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.229550 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.229526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-home\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.229609 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.229551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-tmp-dir\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.229609 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.229592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-model-cache\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.231542 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.231505    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-dshm\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.231805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.231785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2013af00-9176-4429-8d9c-9e977456d921-tls-certs\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.236946 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.236928    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8gkf\" (UniqueName: \"kubernetes.io/projected/2013af00-9176-4429-8d9c-9e977456d921-kube-api-access-n8gkf\") pod \"test-model-dd359c-kserve-6f6bb7995c-4m7lj\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") " pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.280878 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.280851    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:02.552736 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.552707    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"]
Sep 16 07:51:02.553779 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:51:02.553738    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2013af00_9176_4429_8d9c_9e977456d921.slice/crio-5d3f6f7aa9566289c9498b5965ee113bf36eb4747bf6169c40db4fd0b8cc66bb WatchSource:0}: Error finding container 5d3f6f7aa9566289c9498b5965ee113bf36eb4747bf6169c40db4fd0b8cc66bb: Status 404 returned error can't find the container with id 5d3f6f7aa9566289c9498b5965ee113bf36eb4747bf6169c40db4fd0b8cc66bb
Sep 16 07:51:02.737071 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.737040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj" event={"ID":"2013af00-9176-4429-8d9c-9e977456d921","Type":"ContainerStarted","Data":"cf823f8b0cc3d5be4d6791620d54d8ed402acc713a5e86b53319404f755b2641"}
Sep 16 07:51:02.737236 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.737083    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj" event={"ID":"2013af00-9176-4429-8d9c-9e977456d921","Type":"ContainerStarted","Data":"5d3f6f7aa9566289c9498b5965ee113bf36eb4747bf6169c40db4fd0b8cc66bb"}
Sep 16 07:51:02.759359 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:02.759293    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj" podStartSLOduration=1.759271734 podStartE2EDuration="1.759271734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:51:02.75786468 +0000 UTC m=+1023.684476834" watchObservedRunningTime="2026-09-16 07:51:02.759271734 +0000 UTC m=+1023.685883887"
Sep 16 07:51:10.251669 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.251636    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"]
Sep 16 07:51:10.255981 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.255956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.258482 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.258458    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b71faf9e-openshift-default-dockercfg-sdw25\""
Sep 16 07:51:10.265589 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.265568    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"]
Sep 16 07:51:10.311228 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.311198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-envoy\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.311353 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.311237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-credential-socket\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.311353 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.311271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-workload-socket\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.311353 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.311302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-podinfo\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.311353 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.311330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-workload-certs\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.311569 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.311389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-token\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.311569 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.311427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48jgn\" (UniqueName: \"kubernetes.io/projected/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-kube-api-access-48jgn\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.311569 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.311463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istiod-ca-cert\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.311569 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.311487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-data\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.412310 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.412264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-token\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.412502 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.412330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48jgn\" (UniqueName: \"kubernetes.io/projected/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-kube-api-access-48jgn\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.412502 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.412367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istiod-ca-cert\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.412502 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.412401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-data\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.412502 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.412450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-envoy\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.412502 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.412482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-credential-socket\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.412735 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.412518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-workload-socket\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.412735 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.412549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-podinfo\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.412735 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.412602    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-workload-certs\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.413066 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.413013    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-workload-socket\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.413216 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.413092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-workload-certs\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.413292 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.413249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-credential-socket\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.413292 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.413264    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istiod-ca-cert\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.413477 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.413452    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-data\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.415062 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.415039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-envoy\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.415555 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.415537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-podinfo\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.420854 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.420814    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-token\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.421349 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.421322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48jgn\" (UniqueName: \"kubernetes.io/projected/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-kube-api-access-48jgn\") pod \"e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") " pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.568663 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.568604    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:10.903690 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.903660    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"]
Sep 16 07:51:10.904407 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:51:10.904381    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2edd7dd8_6ab7_4d11_886e_dd7b01f43e57.slice/crio-29b5ef45c4cb4c5d0654152c964868c29df8f7ea48ccb2185e7493795e137bb8 WatchSource:0}: Error finding container 29b5ef45c4cb4c5d0654152c964868c29df8f7ea48ccb2185e7493795e137bb8: Status 404 returned error can't find the container with id 29b5ef45c4cb4c5d0654152c964868c29df8f7ea48ccb2185e7493795e137bb8
Sep 16 07:51:10.907266 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.907235    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:51:10.907352 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.907297    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:51:10.907352 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:10.907325    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:51:11.783018 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:11.782976    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz" event={"ID":"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57","Type":"ContainerStarted","Data":"d9891dfce1743c8ae3e6f7c8da271120ad29866005edd7fa59de20079748979c"}
Sep 16 07:51:11.783549 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:11.783028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz" event={"ID":"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57","Type":"ContainerStarted","Data":"29b5ef45c4cb4c5d0654152c964868c29df8f7ea48ccb2185e7493795e137bb8"}
Sep 16 07:51:11.828837 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:11.828782    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz" podStartSLOduration=1.828765567 podStartE2EDuration="1.828765567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:51:11.822864362 +0000 UTC m=+1032.749476514" watchObservedRunningTime="2026-09-16 07:51:11.828765567 +0000 UTC m=+1032.755377720"
Sep 16 07:51:12.281789 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:12.281754    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:12.282019 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:12.281811    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:12.294510 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:12.294479    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:12.569397 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:12.569317    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:12.574423 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:12.574400    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:12.788981 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:12.788948    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:12.789893 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:12.789869    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:12.800176 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:12.800157    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:51:21.038614 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.038578    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"]
Sep 16 07:51:21.042336 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.042315    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"
Sep 16 07:51:21.044766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.044746    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b71faf9e-dockercfg-mnq59\""
Sep 16 07:51:21.044993 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.044978    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b71faf9e\""
Sep 16 07:51:21.053886 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.053862    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"]
Sep 16 07:51:21.117813 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.117782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7wb8\" (UniqueName: \"kubernetes.io/projected/9deac8a6-e833-48d1-ad03-5c68c7cc2221-kube-api-access-v7wb8\") pod \"payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2\" (UID: \"9deac8a6-e833-48d1-ad03-5c68c7cc2221\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"
Sep 16 07:51:21.117978 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.117949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9deac8a6-e833-48d1-ad03-5c68c7cc2221-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2\" (UID: \"9deac8a6-e833-48d1-ad03-5c68c7cc2221\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"
Sep 16 07:51:21.219197 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.219161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7wb8\" (UniqueName: \"kubernetes.io/projected/9deac8a6-e833-48d1-ad03-5c68c7cc2221-kube-api-access-v7wb8\") pod \"payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2\" (UID: \"9deac8a6-e833-48d1-ad03-5c68c7cc2221\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"
Sep 16 07:51:21.219366 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.219235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9deac8a6-e833-48d1-ad03-5c68c7cc2221-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2\" (UID: \"9deac8a6-e833-48d1-ad03-5c68c7cc2221\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"
Sep 16 07:51:21.220322 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.220302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9deac8a6-e833-48d1-ad03-5c68c7cc2221-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2\" (UID: \"9deac8a6-e833-48d1-ad03-5c68c7cc2221\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"
Sep 16 07:51:21.227759 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.227738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7wb8\" (UniqueName: \"kubernetes.io/projected/9deac8a6-e833-48d1-ad03-5c68c7cc2221-kube-api-access-v7wb8\") pod \"payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2\" (UID: \"9deac8a6-e833-48d1-ad03-5c68c7cc2221\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"
Sep 16 07:51:21.354416 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.354350    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"
Sep 16 07:51:21.628457 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.628424    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"]
Sep 16 07:51:21.629915 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:51:21.629883    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9deac8a6_e833_48d1_ad03_5c68c7cc2221.slice/crio-3433edd7c9637f3e379ecf98e7c15182fe4e429f25b69d95a51fe909443acf82 WatchSource:0}: Error finding container 3433edd7c9637f3e379ecf98e7c15182fe4e429f25b69d95a51fe909443acf82: Status 404 returned error can't find the container with id 3433edd7c9637f3e379ecf98e7c15182fe4e429f25b69d95a51fe909443acf82
Sep 16 07:51:21.838577 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.838541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2" event={"ID":"9deac8a6-e833-48d1-ad03-5c68c7cc2221","Type":"ContainerStarted","Data":"e5b9eea0d96d23210fa7c1157b130d4ce45650c6b83f25f877a5b8cf2486af00"}
Sep 16 07:51:21.838760 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.838600    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2" event={"ID":"9deac8a6-e833-48d1-ad03-5c68c7cc2221","Type":"ContainerStarted","Data":"3433edd7c9637f3e379ecf98e7c15182fe4e429f25b69d95a51fe909443acf82"}
Sep 16 07:51:21.838760 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.838709    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"
Sep 16 07:51:21.855735 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:21.855694    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2" podStartSLOduration=0.855680823 podStartE2EDuration="855.680823ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:51:21.854598183 +0000 UTC m=+1042.781210337" watchObservedRunningTime="2026-09-16 07:51:21.855680823 +0000 UTC m=+1042.782292976"
Sep 16 07:51:32.845504 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:32.845473    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"
Sep 16 07:51:40.758495 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:40.758451    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"]
Sep 16 07:51:40.762279 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:40.762254    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:40.765337 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:40.765307    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-82bbbf\"/\"openshift-service-ca.crt\""
Sep 16 07:51:40.765464 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:40.765315    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-82bbbf\"/\"kube-root-ca.crt\""
Sep 16 07:51:40.766369 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:40.766350    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-82bbbf\"/\"default-dockercfg-fqx9x\""
Sep 16 07:51:40.766534 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:40.766517    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-82bbbf\"/\"explicit-model-82bbbf-kserve-self-signed-certs\""
Sep 16 07:51:40.771183 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:40.771118    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"]
Sep 16 07:51:40.914988 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:40.914951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-tmp-dir\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:40.915175 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:40.914998    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7222ab31-056f-425b-984d-27f0fed9e303-tls-certs\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:40.915175 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:40.915024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-dshm\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:40.915175 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:40.915133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g22dl\" (UniqueName: \"kubernetes.io/projected/7222ab31-056f-425b-984d-27f0fed9e303-kube-api-access-g22dl\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:40.915313 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:40.915196    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-home\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:40.915313 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:40.915283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-model-cache\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:41.016170 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.016134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-home\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:41.016344 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.016187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-model-cache\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:41.016344 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.016243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-tmp-dir\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:41.016344 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.016258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7222ab31-056f-425b-984d-27f0fed9e303-tls-certs\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:41.016344 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.016282    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-dshm\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:41.016586 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.016343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g22dl\" (UniqueName: \"kubernetes.io/projected/7222ab31-056f-425b-984d-27f0fed9e303-kube-api-access-g22dl\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:41.016642 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.016610    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-home\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:41.016697 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.016655    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-tmp-dir\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:41.016697 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.016684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-model-cache\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:41.018646 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.018629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-dshm\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:41.018940 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.018915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7222ab31-056f-425b-984d-27f0fed9e303-tls-certs\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:41.027106 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.027047    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g22dl\" (UniqueName: \"kubernetes.io/projected/7222ab31-056f-425b-984d-27f0fed9e303-kube-api-access-g22dl\") pod \"explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") " pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:41.074426 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.074400    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:41.346358 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.346333    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"]
Sep 16 07:51:41.349838 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:51:41.349795    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7222ab31_056f_425b_984d_27f0fed9e303.slice/crio-e4b3836da34d1a4edd04048915b015f901d4216df978bc422a750d5b03664b12 WatchSource:0}: Error finding container e4b3836da34d1a4edd04048915b015f901d4216df978bc422a750d5b03664b12: Status 404 returned error can't find the container with id e4b3836da34d1a4edd04048915b015f901d4216df978bc422a750d5b03664b12
Sep 16 07:51:41.963501 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.963467    2581 generic.go:356] "Generic (PLEG): container finished" podID="2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" containerID="d9891dfce1743c8ae3e6f7c8da271120ad29866005edd7fa59de20079748979c" exitCode=0
Sep 16 07:51:41.963961 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.963540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz" event={"ID":"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57","Type":"ContainerDied","Data":"d9891dfce1743c8ae3e6f7c8da271120ad29866005edd7fa59de20079748979c"}
Sep 16 07:51:41.963961 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.963947    2581 scope.go:117] "RemoveContainer" containerID="d9891dfce1743c8ae3e6f7c8da271120ad29866005edd7fa59de20079748979c"
Sep 16 07:51:41.965204 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.965177    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:51:41.965298 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.965246    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:51:41.965298 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.965263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm" event={"ID":"7222ab31-056f-425b-984d-27f0fed9e303","Type":"ContainerStarted","Data":"f67edd92c319ab9f5eb5769eb980b32cd25eeb6e2903ba1e2b9fb3a40f9a35a5"}
Sep 16 07:51:41.965298 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.965276    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:51:41.965298 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.965293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm" event={"ID":"7222ab31-056f-425b-984d-27f0fed9e303","Type":"ContainerStarted","Data":"e4b3836da34d1a4edd04048915b015f901d4216df978bc422a750d5b03664b12"}
Sep 16 07:51:41.999680 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:41.999607    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm" podStartSLOduration=1.9995917300000001 podStartE2EDuration="1.99959173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:51:41.997721245 +0000 UTC m=+1062.924333397" watchObservedRunningTime="2026-09-16 07:51:41.99959173 +0000 UTC m=+1062.926203882"
Sep 16 07:51:42.789074 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:42.789045    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:42.972243 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:42.972206    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz" event={"ID":"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57","Type":"ContainerStarted","Data":"94d1deb2642de933df6568d7f9ca7da2b91c0bfceb398ce7892710ab6770fd49"}
Sep 16 07:51:43.569305 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:43.569267    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:43.570336 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:43.570314    2581 patch_prober.go:28] interesting pod/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 16 07:51:43.570394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:43.570363    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz" podUID="2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 16 07:51:44.569665 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:44.569629    2581 patch_prober.go:28] interesting pod/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 16 07:51:44.570063 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:44.569684    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz" podUID="2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 16 07:51:45.569297 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:45.569262    2581 patch_prober.go:28] interesting pod/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 16 07:51:45.569471 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:45.569317    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz" podUID="2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 16 07:51:46.569311 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:46.569274    2581 patch_prober.go:28] interesting pod/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 16 07:51:46.569749 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:46.569337    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz" podUID="2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 16 07:51:47.569564 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:47.569528    2581 patch_prober.go:28] interesting pod/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 16 07:51:47.569952 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:47.569580    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz" podUID="2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 16 07:51:48.573187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:48.573157    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:48.573621 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:48.573430    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:48.574136 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:48.574119    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:51:51.074520 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:51.074486    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:51.074936 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:51.074537    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:51.086674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:51.086653    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:52.035203 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:52.035128    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:51:53.873277 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:53.873240    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp"]
Sep 16 07:51:53.876199 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:53.876181    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp"
Sep 16 07:51:53.878566 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:53.878542    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 07:51:53.878688 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:53.878573    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\""
Sep 16 07:51:53.883959 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:53.883747    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp"]
Sep 16 07:51:53.949479 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:53.949450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17157ec1-ca5f-4d04-ab08-45abb7cb0e88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp\" (UID: \"17157ec1-ca5f-4d04-ab08-45abb7cb0e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp"
Sep 16 07:51:54.051053 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:54.051025    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17157ec1-ca5f-4d04-ab08-45abb7cb0e88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp\" (UID: \"17157ec1-ca5f-4d04-ab08-45abb7cb0e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp"
Sep 16 07:51:54.051734 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:54.051714    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17157ec1-ca5f-4d04-ab08-45abb7cb0e88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp\" (UID: \"17157ec1-ca5f-4d04-ab08-45abb7cb0e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp"
Sep 16 07:51:54.188269 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:54.188239    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp"
Sep 16 07:51:54.454515 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:54.454492    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp"]
Sep 16 07:51:54.456986 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:51:54.456957    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17157ec1_ca5f_4d04_ab08_45abb7cb0e88.slice/crio-f87476716b32f4b61db3bf30f72c36130600506394858a23ec618d7f9fb45f4a WatchSource:0}: Error finding container f87476716b32f4b61db3bf30f72c36130600506394858a23ec618d7f9fb45f4a: Status 404 returned error can't find the container with id f87476716b32f4b61db3bf30f72c36130600506394858a23ec618d7f9fb45f4a
Sep 16 07:51:55.041556 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:55.041471    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp" event={"ID":"17157ec1-ca5f-4d04-ab08-45abb7cb0e88","Type":"ContainerStarted","Data":"3fb106a3b6d86215ab344d71beb4ca4e2b6529b3c498b31e8dbe09290f620a4c"}
Sep 16 07:51:55.041556 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:55.041534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp" event={"ID":"17157ec1-ca5f-4d04-ab08-45abb7cb0e88","Type":"ContainerStarted","Data":"f87476716b32f4b61db3bf30f72c36130600506394858a23ec618d7f9fb45f4a"}
Sep 16 07:51:55.057248 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:55.057193    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp" podStartSLOduration=2.057173546 podStartE2EDuration="2.057173546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:51:55.056962691 +0000 UTC m=+1075.983574843" watchObservedRunningTime="2026-09-16 07:51:55.057173546 +0000 UTC m=+1075.983785698"
Sep 16 07:51:56.048707 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:56.048670    2581 generic.go:356] "Generic (PLEG): container finished" podID="17157ec1-ca5f-4d04-ab08-45abb7cb0e88" containerID="3fb106a3b6d86215ab344d71beb4ca4e2b6529b3c498b31e8dbe09290f620a4c" exitCode=0
Sep 16 07:51:56.049095 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:56.048725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp" event={"ID":"17157ec1-ca5f-4d04-ab08-45abb7cb0e88","Type":"ContainerDied","Data":"3fb106a3b6d86215ab344d71beb4ca4e2b6529b3c498b31e8dbe09290f620a4c"}
Sep 16 07:51:57.189845 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:57.189809    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp"
Sep 16 07:51:57.282910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:57.282879    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17157ec1-ca5f-4d04-ab08-45abb7cb0e88-maas-api-service-ca\") pod \"17157ec1-ca5f-4d04-ab08-45abb7cb0e88\" (UID: \"17157ec1-ca5f-4d04-ab08-45abb7cb0e88\") "
Sep 16 07:51:57.283266 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:57.283233    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17157ec1-ca5f-4d04-ab08-45abb7cb0e88-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17157ec1-ca5f-4d04-ab08-45abb7cb0e88" (UID: "17157ec1-ca5f-4d04-ab08-45abb7cb0e88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:51:57.384480 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:57.384420    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17157ec1-ca5f-4d04-ab08-45abb7cb0e88-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:51:58.061098 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:58.061017    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp"
Sep 16 07:51:58.061098 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:58.061040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp" event={"ID":"17157ec1-ca5f-4d04-ab08-45abb7cb0e88","Type":"ContainerDied","Data":"f87476716b32f4b61db3bf30f72c36130600506394858a23ec618d7f9fb45f4a"}
Sep 16 07:51:58.061098 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:51:58.061089    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f87476716b32f4b61db3bf30f72c36130600506394858a23ec618d7f9fb45f4a"
Sep 16 07:52:04.566081 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:04.566047    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"]
Sep 16 07:52:04.566580 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:04.566271    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2" podUID="9deac8a6-e833-48d1-ad03-5c68c7cc2221" containerName="payload-pre-processing" containerID="cri-o://e5b9eea0d96d23210fa7c1157b130d4ce45650c6b83f25f877a5b8cf2486af00" gracePeriod=30
Sep 16 07:52:04.718213 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:04.718177    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp"]
Sep 16 07:52:04.720452 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:04.720427    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b71faf9e-b6prp"]
Sep 16 07:52:04.857265 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:04.857235    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"
Sep 16 07:52:04.958908 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:04.958867    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7wb8\" (UniqueName: \"kubernetes.io/projected/9deac8a6-e833-48d1-ad03-5c68c7cc2221-kube-api-access-v7wb8\") pod \"9deac8a6-e833-48d1-ad03-5c68c7cc2221\" (UID: \"9deac8a6-e833-48d1-ad03-5c68c7cc2221\") "
Sep 16 07:52:04.959089 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:04.959015    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9deac8a6-e833-48d1-ad03-5c68c7cc2221-plugins-config-volume\") pod \"9deac8a6-e833-48d1-ad03-5c68c7cc2221\" (UID: \"9deac8a6-e833-48d1-ad03-5c68c7cc2221\") "
Sep 16 07:52:04.959397 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:04.959355    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9deac8a6-e833-48d1-ad03-5c68c7cc2221-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9deac8a6-e833-48d1-ad03-5c68c7cc2221" (UID: "9deac8a6-e833-48d1-ad03-5c68c7cc2221"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:52:04.961227 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:04.961189    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9deac8a6-e833-48d1-ad03-5c68c7cc2221-kube-api-access-v7wb8" (OuterVolumeSpecName: "kube-api-access-v7wb8") pod "9deac8a6-e833-48d1-ad03-5c68c7cc2221" (UID: "9deac8a6-e833-48d1-ad03-5c68c7cc2221"). InnerVolumeSpecName "kube-api-access-v7wb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:52:05.060353 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:05.060319    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7wb8\" (UniqueName: \"kubernetes.io/projected/9deac8a6-e833-48d1-ad03-5c68c7cc2221-kube-api-access-v7wb8\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:05.060353 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:05.060348    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9deac8a6-e833-48d1-ad03-5c68c7cc2221-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:05.097890 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:05.097779    2581 generic.go:356] "Generic (PLEG): container finished" podID="9deac8a6-e833-48d1-ad03-5c68c7cc2221" containerID="e5b9eea0d96d23210fa7c1157b130d4ce45650c6b83f25f877a5b8cf2486af00" exitCode=0
Sep 16 07:52:05.097890 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:05.097847    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"
Sep 16 07:52:05.098064 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:05.097896    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2" event={"ID":"9deac8a6-e833-48d1-ad03-5c68c7cc2221","Type":"ContainerDied","Data":"e5b9eea0d96d23210fa7c1157b130d4ce45650c6b83f25f877a5b8cf2486af00"}
Sep 16 07:52:05.098064 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:05.097944    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2" event={"ID":"9deac8a6-e833-48d1-ad03-5c68c7cc2221","Type":"ContainerDied","Data":"3433edd7c9637f3e379ecf98e7c15182fe4e429f25b69d95a51fe909443acf82"}
Sep 16 07:52:05.098064 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:05.097965    2581 scope.go:117] "RemoveContainer" containerID="e5b9eea0d96d23210fa7c1157b130d4ce45650c6b83f25f877a5b8cf2486af00"
Sep 16 07:52:05.116039 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:05.116016    2581 scope.go:117] "RemoveContainer" containerID="e5b9eea0d96d23210fa7c1157b130d4ce45650c6b83f25f877a5b8cf2486af00"
Sep 16 07:52:05.116341 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:52:05.116316    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b9eea0d96d23210fa7c1157b130d4ce45650c6b83f25f877a5b8cf2486af00\": container with ID starting with e5b9eea0d96d23210fa7c1157b130d4ce45650c6b83f25f877a5b8cf2486af00 not found: ID does not exist" containerID="e5b9eea0d96d23210fa7c1157b130d4ce45650c6b83f25f877a5b8cf2486af00"
Sep 16 07:52:05.116435 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:05.116347    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b9eea0d96d23210fa7c1157b130d4ce45650c6b83f25f877a5b8cf2486af00"} err="failed to get container status \"e5b9eea0d96d23210fa7c1157b130d4ce45650c6b83f25f877a5b8cf2486af00\": rpc error: code = NotFound desc = could not find container \"e5b9eea0d96d23210fa7c1157b130d4ce45650c6b83f25f877a5b8cf2486af00\": container with ID starting with e5b9eea0d96d23210fa7c1157b130d4ce45650c6b83f25f877a5b8cf2486af00 not found: ID does not exist"
Sep 16 07:52:05.134709 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:05.134680    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"]
Sep 16 07:52:05.146267 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:05.146247    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b71faf9e-6c9685c64b-gz5w2"]
Sep 16 07:52:05.684259 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:05.684230    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17157ec1-ca5f-4d04-ab08-45abb7cb0e88" path="/var/lib/kubelet/pods/17157ec1-ca5f-4d04-ab08-45abb7cb0e88/volumes"
Sep 16 07:52:05.684621 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:05.684585    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9deac8a6-e833-48d1-ad03-5c68c7cc2221" path="/var/lib/kubelet/pods/9deac8a6-e833-48d1-ad03-5c68c7cc2221/volumes"
Sep 16 07:52:11.237172 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:11.237134    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"]
Sep 16 07:52:11.237639 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:11.237402    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz" podUID="2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" containerName="istio-proxy" containerID="cri-o://94d1deb2642de933df6568d7f9ca7da2b91c0bfceb398ce7892710ab6770fd49" gracePeriod=30
Sep 16 07:52:16.494649 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.494613    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:52:16.574616 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.574529    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-podinfo\") pod \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") "
Sep 16 07:52:16.574616 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.574581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-credential-socket\") pod \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") "
Sep 16 07:52:16.574616 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.574601    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-token\") pod \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") "
Sep 16 07:52:16.574910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.574625    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-workload-socket\") pod \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") "
Sep 16 07:52:16.574910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.574702    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-workload-certs\") pod \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") "
Sep 16 07:52:16.574910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.574726    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48jgn\" (UniqueName: \"kubernetes.io/projected/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-kube-api-access-48jgn\") pod \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") "
Sep 16 07:52:16.574910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.574759    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istiod-ca-cert\") pod \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") "
Sep 16 07:52:16.574910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.574785    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-data\") pod \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") "
Sep 16 07:52:16.574910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.574871    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" (UID: "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:52:16.574910 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.574899    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-envoy\") pod \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\" (UID: \"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57\") "
Sep 16 07:52:16.575297 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.575057    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-data" (OuterVolumeSpecName: "istio-data") pod "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" (UID: "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:52:16.575297 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.575120    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" (UID: "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:52:16.575297 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.575154    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" (UID: "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:52:16.575297 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.575170    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" (UID: "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:52:16.575516 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.575379    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:16.575516 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.575400    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:16.575516 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.575414    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:16.575516 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.575426    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:16.575516 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.575440    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:16.577001 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.576966    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" (UID: "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:52:16.577112 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.577031    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-token" (OuterVolumeSpecName: "istio-token") pod "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" (UID: "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:52:16.577112 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.577066    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" (UID: "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 07:52:16.577112 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.577067    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-kube-api-access-48jgn" (OuterVolumeSpecName: "kube-api-access-48jgn") pod "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" (UID: "2edd7dd8-6ab7-4d11-886e-dd7b01f43e57"). InnerVolumeSpecName "kube-api-access-48jgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:52:16.676280 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.676250    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:16.676280 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.676276    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:16.676280 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.676285    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:16.676504 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:16.676295    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48jgn\" (UniqueName: \"kubernetes.io/projected/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57-kube-api-access-48jgn\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:17.169530 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:17.169497    2581 generic.go:356] "Generic (PLEG): container finished" podID="2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" containerID="94d1deb2642de933df6568d7f9ca7da2b91c0bfceb398ce7892710ab6770fd49" exitCode=0
Sep 16 07:52:17.169765 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:17.169550    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"
Sep 16 07:52:17.169765 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:17.169586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz" event={"ID":"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57","Type":"ContainerDied","Data":"94d1deb2642de933df6568d7f9ca7da2b91c0bfceb398ce7892710ab6770fd49"}
Sep 16 07:52:17.169765 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:17.169652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz" event={"ID":"2edd7dd8-6ab7-4d11-886e-dd7b01f43e57","Type":"ContainerDied","Data":"29b5ef45c4cb4c5d0654152c964868c29df8f7ea48ccb2185e7493795e137bb8"}
Sep 16 07:52:17.169765 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:17.169684    2581 scope.go:117] "RemoveContainer" containerID="94d1deb2642de933df6568d7f9ca7da2b91c0bfceb398ce7892710ab6770fd49"
Sep 16 07:52:17.180069 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:17.180046    2581 scope.go:117] "RemoveContainer" containerID="d9891dfce1743c8ae3e6f7c8da271120ad29866005edd7fa59de20079748979c"
Sep 16 07:52:17.188861 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:17.188840    2581 scope.go:117] "RemoveContainer" containerID="94d1deb2642de933df6568d7f9ca7da2b91c0bfceb398ce7892710ab6770fd49"
Sep 16 07:52:17.189152 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:52:17.189132    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d1deb2642de933df6568d7f9ca7da2b91c0bfceb398ce7892710ab6770fd49\": container with ID starting with 94d1deb2642de933df6568d7f9ca7da2b91c0bfceb398ce7892710ab6770fd49 not found: ID does not exist" containerID="94d1deb2642de933df6568d7f9ca7da2b91c0bfceb398ce7892710ab6770fd49"
Sep 16 07:52:17.189207 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:17.189163    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d1deb2642de933df6568d7f9ca7da2b91c0bfceb398ce7892710ab6770fd49"} err="failed to get container status \"94d1deb2642de933df6568d7f9ca7da2b91c0bfceb398ce7892710ab6770fd49\": rpc error: code = NotFound desc = could not find container \"94d1deb2642de933df6568d7f9ca7da2b91c0bfceb398ce7892710ab6770fd49\": container with ID starting with 94d1deb2642de933df6568d7f9ca7da2b91c0bfceb398ce7892710ab6770fd49 not found: ID does not exist"
Sep 16 07:52:17.189207 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:17.189183    2581 scope.go:117] "RemoveContainer" containerID="d9891dfce1743c8ae3e6f7c8da271120ad29866005edd7fa59de20079748979c"
Sep 16 07:52:17.189433 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:52:17.189413    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9891dfce1743c8ae3e6f7c8da271120ad29866005edd7fa59de20079748979c\": container with ID starting with d9891dfce1743c8ae3e6f7c8da271120ad29866005edd7fa59de20079748979c not found: ID does not exist" containerID="d9891dfce1743c8ae3e6f7c8da271120ad29866005edd7fa59de20079748979c"
Sep 16 07:52:17.189487 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:17.189440    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9891dfce1743c8ae3e6f7c8da271120ad29866005edd7fa59de20079748979c"} err="failed to get container status \"d9891dfce1743c8ae3e6f7c8da271120ad29866005edd7fa59de20079748979c\": rpc error: code = NotFound desc = could not find container \"d9891dfce1743c8ae3e6f7c8da271120ad29866005edd7fa59de20079748979c\": container with ID starting with d9891dfce1743c8ae3e6f7c8da271120ad29866005edd7fa59de20079748979c not found: ID does not exist"
Sep 16 07:52:17.195670 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:17.195638    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"]
Sep 16 07:52:17.201116 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:17.201091    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b71faf9e-openshift-default-86bbf78f84-l79pz"]
Sep 16 07:52:17.684546 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:17.684513    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" path="/var/lib/kubelet/pods/2edd7dd8-6ab7-4d11-886e-dd7b01f43e57/volumes"
Sep 16 07:52:19.497058 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:19.497027    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"]
Sep 16 07:52:23.644141 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:23.644107    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"]
Sep 16 07:52:23.644746 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:23.644398    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl" podUID="dadab9e0-00ff-40c8-ad4b-788e17742890" containerName="payload-processing" containerID="cri-o://587c328cd3c84336a223d70148d3928625dda54e583193a020b1cf9421ed7be7" gracePeriod=30
Sep 16 07:52:23.648415 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:23.648388    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"]
Sep 16 07:52:23.649109 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:23.648753    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6" podUID="afe3901d-6928-46d9-9f7b-c7f4a5b0f599" containerName="payload-pre-processing" containerID="cri-o://e058cf3a8fe9cd13caf3dc37b3dee30c74092076c14da45cb7b23f1db0588d90" gracePeriod=30
Sep 16 07:52:23.957066 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:23.957028    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"
Sep 16 07:52:23.977165 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:23.977139    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"
Sep 16 07:52:24.052320 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.052288    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wwnl\" (UniqueName: \"kubernetes.io/projected/afe3901d-6928-46d9-9f7b-c7f4a5b0f599-kube-api-access-7wwnl\") pod \"afe3901d-6928-46d9-9f7b-c7f4a5b0f599\" (UID: \"afe3901d-6928-46d9-9f7b-c7f4a5b0f599\") "
Sep 16 07:52:24.052493 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.052370    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe3901d-6928-46d9-9f7b-c7f4a5b0f599-plugins-config-volume\") pod \"afe3901d-6928-46d9-9f7b-c7f4a5b0f599\" (UID: \"afe3901d-6928-46d9-9f7b-c7f4a5b0f599\") "
Sep 16 07:52:24.052493 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.052396    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dadab9e0-00ff-40c8-ad4b-788e17742890-plugins-config-volume\") pod \"dadab9e0-00ff-40c8-ad4b-788e17742890\" (UID: \"dadab9e0-00ff-40c8-ad4b-788e17742890\") "
Sep 16 07:52:24.052493 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.052445    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxc66\" (UniqueName: \"kubernetes.io/projected/dadab9e0-00ff-40c8-ad4b-788e17742890-kube-api-access-qxc66\") pod \"dadab9e0-00ff-40c8-ad4b-788e17742890\" (UID: \"dadab9e0-00ff-40c8-ad4b-788e17742890\") "
Sep 16 07:52:24.052815 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.052778    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe3901d-6928-46d9-9f7b-c7f4a5b0f599-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "afe3901d-6928-46d9-9f7b-c7f4a5b0f599" (UID: "afe3901d-6928-46d9-9f7b-c7f4a5b0f599"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:52:24.052952 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.052788    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dadab9e0-00ff-40c8-ad4b-788e17742890-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dadab9e0-00ff-40c8-ad4b-788e17742890" (UID: "dadab9e0-00ff-40c8-ad4b-788e17742890"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:52:24.054689 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.054667    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe3901d-6928-46d9-9f7b-c7f4a5b0f599-kube-api-access-7wwnl" (OuterVolumeSpecName: "kube-api-access-7wwnl") pod "afe3901d-6928-46d9-9f7b-c7f4a5b0f599" (UID: "afe3901d-6928-46d9-9f7b-c7f4a5b0f599"). InnerVolumeSpecName "kube-api-access-7wwnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:52:24.054923 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.054907    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dadab9e0-00ff-40c8-ad4b-788e17742890-kube-api-access-qxc66" (OuterVolumeSpecName: "kube-api-access-qxc66") pod "dadab9e0-00ff-40c8-ad4b-788e17742890" (UID: "dadab9e0-00ff-40c8-ad4b-788e17742890"). InnerVolumeSpecName "kube-api-access-qxc66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:52:24.153627 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.153595    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wwnl\" (UniqueName: \"kubernetes.io/projected/afe3901d-6928-46d9-9f7b-c7f4a5b0f599-kube-api-access-7wwnl\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:24.153627 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.153624    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe3901d-6928-46d9-9f7b-c7f4a5b0f599-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:24.153627 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.153633    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dadab9e0-00ff-40c8-ad4b-788e17742890-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:24.153894 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.153642    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxc66\" (UniqueName: \"kubernetes.io/projected/dadab9e0-00ff-40c8-ad4b-788e17742890-kube-api-access-qxc66\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:24.214893 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.214782    2581 generic.go:356] "Generic (PLEG): container finished" podID="afe3901d-6928-46d9-9f7b-c7f4a5b0f599" containerID="e058cf3a8fe9cd13caf3dc37b3dee30c74092076c14da45cb7b23f1db0588d90" exitCode=0
Sep 16 07:52:24.214893 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.214864    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"
Sep 16 07:52:24.215109 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.214858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6" event={"ID":"afe3901d-6928-46d9-9f7b-c7f4a5b0f599","Type":"ContainerDied","Data":"e058cf3a8fe9cd13caf3dc37b3dee30c74092076c14da45cb7b23f1db0588d90"}
Sep 16 07:52:24.215109 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.215020    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6" event={"ID":"afe3901d-6928-46d9-9f7b-c7f4a5b0f599","Type":"ContainerDied","Data":"69b4525fb88a2834d5e7b33fbafdc7f0be51d2d2f774123ad55ee0790587aa1f"}
Sep 16 07:52:24.215109 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.215054    2581 scope.go:117] "RemoveContainer" containerID="e058cf3a8fe9cd13caf3dc37b3dee30c74092076c14da45cb7b23f1db0588d90"
Sep 16 07:52:24.216379 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.216358    2581 generic.go:356] "Generic (PLEG): container finished" podID="dadab9e0-00ff-40c8-ad4b-788e17742890" containerID="587c328cd3c84336a223d70148d3928625dda54e583193a020b1cf9421ed7be7" exitCode=0
Sep 16 07:52:24.216468 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.216392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl" event={"ID":"dadab9e0-00ff-40c8-ad4b-788e17742890","Type":"ContainerDied","Data":"587c328cd3c84336a223d70148d3928625dda54e583193a020b1cf9421ed7be7"}
Sep 16 07:52:24.216468 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.216445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl" event={"ID":"dadab9e0-00ff-40c8-ad4b-788e17742890","Type":"ContainerDied","Data":"ded1e7a7e26c9a382ab0c3be5f59095adc07f31256638019af2cb5fc5f5f578f"}
Sep 16 07:52:24.216559 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.216402    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"
Sep 16 07:52:24.225251 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.225232    2581 scope.go:117] "RemoveContainer" containerID="e058cf3a8fe9cd13caf3dc37b3dee30c74092076c14da45cb7b23f1db0588d90"
Sep 16 07:52:24.225733 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:52:24.225647    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e058cf3a8fe9cd13caf3dc37b3dee30c74092076c14da45cb7b23f1db0588d90\": container with ID starting with e058cf3a8fe9cd13caf3dc37b3dee30c74092076c14da45cb7b23f1db0588d90 not found: ID does not exist" containerID="e058cf3a8fe9cd13caf3dc37b3dee30c74092076c14da45cb7b23f1db0588d90"
Sep 16 07:52:24.225733 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.225679    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e058cf3a8fe9cd13caf3dc37b3dee30c74092076c14da45cb7b23f1db0588d90"} err="failed to get container status \"e058cf3a8fe9cd13caf3dc37b3dee30c74092076c14da45cb7b23f1db0588d90\": rpc error: code = NotFound desc = could not find container \"e058cf3a8fe9cd13caf3dc37b3dee30c74092076c14da45cb7b23f1db0588d90\": container with ID starting with e058cf3a8fe9cd13caf3dc37b3dee30c74092076c14da45cb7b23f1db0588d90 not found: ID does not exist"
Sep 16 07:52:24.225733 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.225700    2581 scope.go:117] "RemoveContainer" containerID="587c328cd3c84336a223d70148d3928625dda54e583193a020b1cf9421ed7be7"
Sep 16 07:52:24.234710 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.234692    2581 scope.go:117] "RemoveContainer" containerID="587c328cd3c84336a223d70148d3928625dda54e583193a020b1cf9421ed7be7"
Sep 16 07:52:24.235039 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:52:24.235021    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"587c328cd3c84336a223d70148d3928625dda54e583193a020b1cf9421ed7be7\": container with ID starting with 587c328cd3c84336a223d70148d3928625dda54e583193a020b1cf9421ed7be7 not found: ID does not exist" containerID="587c328cd3c84336a223d70148d3928625dda54e583193a020b1cf9421ed7be7"
Sep 16 07:52:24.235102 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.235045    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"587c328cd3c84336a223d70148d3928625dda54e583193a020b1cf9421ed7be7"} err="failed to get container status \"587c328cd3c84336a223d70148d3928625dda54e583193a020b1cf9421ed7be7\": rpc error: code = NotFound desc = could not find container \"587c328cd3c84336a223d70148d3928625dda54e583193a020b1cf9421ed7be7\": container with ID starting with 587c328cd3c84336a223d70148d3928625dda54e583193a020b1cf9421ed7be7 not found: ID does not exist"
Sep 16 07:52:24.246615 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.246595    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"]
Sep 16 07:52:24.251240 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.251220    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-82bbbf-667f9bb857-jgjzl"]
Sep 16 07:52:24.263279 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.263255    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"]
Sep 16 07:52:24.268418 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:24.268396    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-82bbbf-fb77d7f78-wwws6"]
Sep 16 07:52:25.684238 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.684207    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afe3901d-6928-46d9-9f7b-c7f4a5b0f599" path="/var/lib/kubelet/pods/afe3901d-6928-46d9-9f7b-c7f4a5b0f599/volumes"
Sep 16 07:52:25.684605 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.684571    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dadab9e0-00ff-40c8-ad4b-788e17742890" path="/var/lib/kubelet/pods/dadab9e0-00ff-40c8-ad4b-788e17742890/volumes"
Sep 16 07:52:25.975019 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.974923    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"]
Sep 16 07:52:25.975438 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975419    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afe3901d-6928-46d9-9f7b-c7f4a5b0f599" containerName="payload-pre-processing"
Sep 16 07:52:25.975551 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975441    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="afe3901d-6928-46d9-9f7b-c7f4a5b0f599" containerName="payload-pre-processing"
Sep 16 07:52:25.975551 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975456    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dadab9e0-00ff-40c8-ad4b-788e17742890" containerName="payload-processing"
Sep 16 07:52:25.975551 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975464    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dadab9e0-00ff-40c8-ad4b-788e17742890" containerName="payload-processing"
Sep 16 07:52:25.975551 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975473    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17157ec1-ca5f-4d04-ab08-45abb7cb0e88" containerName="revoke-keys"
Sep 16 07:52:25.975551 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975482    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="17157ec1-ca5f-4d04-ab08-45abb7cb0e88" containerName="revoke-keys"
Sep 16 07:52:25.975551 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975504    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" containerName="istio-proxy"
Sep 16 07:52:25.975551 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975512    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" containerName="istio-proxy"
Sep 16 07:52:25.975551 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975527    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" containerName="istio-proxy"
Sep 16 07:52:25.975551 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975535    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" containerName="istio-proxy"
Sep 16 07:52:25.975551 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975553    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9deac8a6-e833-48d1-ad03-5c68c7cc2221" containerName="payload-pre-processing"
Sep 16 07:52:25.976091 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975561    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9deac8a6-e833-48d1-ad03-5c68c7cc2221" containerName="payload-pre-processing"
Sep 16 07:52:25.976091 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975694    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="17157ec1-ca5f-4d04-ab08-45abb7cb0e88" containerName="revoke-keys"
Sep 16 07:52:25.976091 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975709    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9deac8a6-e833-48d1-ad03-5c68c7cc2221" containerName="payload-pre-processing"
Sep 16 07:52:25.976091 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975718    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" containerName="istio-proxy"
Sep 16 07:52:25.976091 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975732    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="afe3901d-6928-46d9-9f7b-c7f4a5b0f599" containerName="payload-pre-processing"
Sep 16 07:52:25.976091 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.975743    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dadab9e0-00ff-40c8-ad4b-788e17742890" containerName="payload-processing"
Sep 16 07:52:25.980608 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.980587    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:25.983152 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.983131    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-4a243c-openshift-default-dockercfg-ttc9p\""
Sep 16 07:52:25.992228 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:25.992164    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"]
Sep 16 07:52:26.169738 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.169695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3418d981-b15f-4ea2-a807-75b731ea0804-istiod-ca-cert\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.169738 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.169743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3418d981-b15f-4ea2-a807-75b731ea0804-istio-podinfo\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.170023 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.169782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-istio-envoy\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.170023 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.169810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-workload-socket\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.170023 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.169909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-credential-socket\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.170023 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.169939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vk4v\" (UniqueName: \"kubernetes.io/projected/3418d981-b15f-4ea2-a807-75b731ea0804-kube-api-access-4vk4v\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.170023 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.169960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-istio-data\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.170023 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.169981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-workload-certs\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.170219 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.170078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3418d981-b15f-4ea2-a807-75b731ea0804-istio-token\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.271073 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.270979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-istio-envoy\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.271073 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.271026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-workload-socket\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.271312 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.271108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-credential-socket\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.271312 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.271147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vk4v\" (UniqueName: \"kubernetes.io/projected/3418d981-b15f-4ea2-a807-75b731ea0804-kube-api-access-4vk4v\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.271312 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.271176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-istio-data\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.271312 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.271206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-workload-certs\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.271312 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.271246    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3418d981-b15f-4ea2-a807-75b731ea0804-istio-token\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.271556 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.271333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3418d981-b15f-4ea2-a807-75b731ea0804-istiod-ca-cert\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.271556 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.271372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3418d981-b15f-4ea2-a807-75b731ea0804-istio-podinfo\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.271556 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.271465    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-workload-socket\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.271556 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.271516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-credential-socket\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.271780 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.271760    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-workload-certs\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.271863 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.271799    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-istio-data\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.272237 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.272216    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3418d981-b15f-4ea2-a807-75b731ea0804-istiod-ca-cert\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.273236 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.273215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-istio-envoy\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.273833 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.273804    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3418d981-b15f-4ea2-a807-75b731ea0804-istio-podinfo\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.280292 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.280260    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vk4v\" (UniqueName: \"kubernetes.io/projected/3418d981-b15f-4ea2-a807-75b731ea0804-kube-api-access-4vk4v\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.281051 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.281031    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3418d981-b15f-4ea2-a807-75b731ea0804-istio-token\") pod \"e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") " pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.294996 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.294974    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:26.578327 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.578302    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"]
Sep 16 07:52:26.580303 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:52:26.580271    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3418d981_b15f_4ea2_a807_75b731ea0804.slice/crio-a199be26fe5f91364c82f181e9fad5d336d0eec39b4eb536aa66a342923196a1 WatchSource:0}: Error finding container a199be26fe5f91364c82f181e9fad5d336d0eec39b4eb536aa66a342923196a1: Status 404 returned error can't find the container with id a199be26fe5f91364c82f181e9fad5d336d0eec39b4eb536aa66a342923196a1
Sep 16 07:52:26.582611 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.582566    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:52:26.582691 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.582671    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:52:26.582741 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:26.582718    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:52:27.237367 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:27.237328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l" event={"ID":"3418d981-b15f-4ea2-a807-75b731ea0804","Type":"ContainerStarted","Data":"795f0b061a10bd49bbf7833f8f1dc62bd052786080320164024a9e86d5600d7d"}
Sep 16 07:52:27.237735 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:27.237394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l" event={"ID":"3418d981-b15f-4ea2-a807-75b731ea0804","Type":"ContainerStarted","Data":"a199be26fe5f91364c82f181e9fad5d336d0eec39b4eb536aa66a342923196a1"}
Sep 16 07:52:27.258795 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:27.258751    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l" podStartSLOduration=2.258736867 podStartE2EDuration="2.258736867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:52:27.256078859 +0000 UTC m=+1108.182691009" watchObservedRunningTime="2026-09-16 07:52:27.258736867 +0000 UTC m=+1108.185349020"
Sep 16 07:52:27.295152 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:27.295124    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:27.300033 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:27.300002    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:28.248431 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:28.248400    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:28.249446 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:28.249428    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:33.118568 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:33.118533    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"]
Sep 16 07:52:33.119162 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:33.119143    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2edd7dd8-6ab7-4d11-886e-dd7b01f43e57" containerName="istio-proxy"
Sep 16 07:52:33.122791 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:33.122772    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"
Sep 16 07:52:33.126956 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:33.126902    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-4a243c\""
Sep 16 07:52:33.128401 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:33.128382    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-4a243c-dockercfg-n7fr9\""
Sep 16 07:52:33.131692 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:33.131668    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"]
Sep 16 07:52:33.237550 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:33.237500    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k6r5\" (UniqueName: \"kubernetes.io/projected/79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e-kube-api-access-5k6r5\") pod \"payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd\" (UID: \"79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"
Sep 16 07:52:33.237723 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:33.237570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd\" (UID: \"79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"
Sep 16 07:52:33.338669 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:33.338632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k6r5\" (UniqueName: \"kubernetes.io/projected/79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e-kube-api-access-5k6r5\") pod \"payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd\" (UID: \"79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"
Sep 16 07:52:33.338669 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:33.338670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd\" (UID: \"79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"
Sep 16 07:52:33.339333 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:33.339312    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd\" (UID: \"79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"
Sep 16 07:52:33.347512 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:33.347479    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k6r5\" (UniqueName: \"kubernetes.io/projected/79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e-kube-api-access-5k6r5\") pod \"payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd\" (UID: \"79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"
Sep 16 07:52:33.436030 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:33.435993    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"
Sep 16 07:52:33.706437 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:33.706409    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"]
Sep 16 07:52:33.707670 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:52:33.707648    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bfb54d_8e46_4fd5_b881_a4ec45ebfd7e.slice/crio-f492fdc8e553877f62db8671a7872d3f336ec62d5f760636a757848f08cbe0a4 WatchSource:0}: Error finding container f492fdc8e553877f62db8671a7872d3f336ec62d5f760636a757848f08cbe0a4: Status 404 returned error can't find the container with id f492fdc8e553877f62db8671a7872d3f336ec62d5f760636a757848f08cbe0a4
Sep 16 07:52:34.282800 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.282761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd" event={"ID":"79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e","Type":"ContainerStarted","Data":"1ed92a2896c8e4f1294cbe677f3bfe54fba4258981023ecab5073f011f043c18"}
Sep 16 07:52:34.283226 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.282838    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd" event={"ID":"79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e","Type":"ContainerStarted","Data":"f492fdc8e553877f62db8671a7872d3f336ec62d5f760636a757848f08cbe0a4"}
Sep 16 07:52:34.283226 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.282997    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"
Sep 16 07:52:34.302815 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.302753    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd" podStartSLOduration=1.302737394 podStartE2EDuration="1.302737394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:52:34.301644839 +0000 UTC m=+1115.228256991" watchObservedRunningTime="2026-09-16 07:52:34.302737394 +0000 UTC m=+1115.229349548"
Sep 16 07:52:34.516288 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.516241    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm" podUID="7222ab31-056f-425b-984d-27f0fed9e303" containerName="main" containerID="cri-o://f67edd92c319ab9f5eb5769eb980b32cd25eeb6e2903ba1e2b9fb3a40f9a35a5" gracePeriod=45
Sep 16 07:52:34.775083 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.775061    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:52:34.852766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.852727    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-tmp-dir\") pod \"7222ab31-056f-425b-984d-27f0fed9e303\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") "
Sep 16 07:52:34.852766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.852757    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-home\") pod \"7222ab31-056f-425b-984d-27f0fed9e303\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") "
Sep 16 07:52:34.852949 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.852787    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-model-cache\") pod \"7222ab31-056f-425b-984d-27f0fed9e303\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") "
Sep 16 07:52:34.852949 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.852811    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7222ab31-056f-425b-984d-27f0fed9e303-tls-certs\") pod \"7222ab31-056f-425b-984d-27f0fed9e303\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") "
Sep 16 07:52:34.852949 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.852905    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g22dl\" (UniqueName: \"kubernetes.io/projected/7222ab31-056f-425b-984d-27f0fed9e303-kube-api-access-g22dl\") pod \"7222ab31-056f-425b-984d-27f0fed9e303\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") "
Sep 16 07:52:34.853101 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.852996    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-dshm\") pod \"7222ab31-056f-425b-984d-27f0fed9e303\" (UID: \"7222ab31-056f-425b-984d-27f0fed9e303\") "
Sep 16 07:52:34.853101 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.853028    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7222ab31-056f-425b-984d-27f0fed9e303" (UID: "7222ab31-056f-425b-984d-27f0fed9e303"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:52:34.853101 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.853035    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-home" (OuterVolumeSpecName: "home") pod "7222ab31-056f-425b-984d-27f0fed9e303" (UID: "7222ab31-056f-425b-984d-27f0fed9e303"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:52:34.853101 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.853056    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-model-cache" (OuterVolumeSpecName: "model-cache") pod "7222ab31-056f-425b-984d-27f0fed9e303" (UID: "7222ab31-056f-425b-984d-27f0fed9e303"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:52:34.853318 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.853282    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:34.853318 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.853295    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:34.853318 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.853303    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:34.855108 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.855081    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-dshm" (OuterVolumeSpecName: "dshm") pod "7222ab31-056f-425b-984d-27f0fed9e303" (UID: "7222ab31-056f-425b-984d-27f0fed9e303"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:52:34.855206 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.855160    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7222ab31-056f-425b-984d-27f0fed9e303-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7222ab31-056f-425b-984d-27f0fed9e303" (UID: "7222ab31-056f-425b-984d-27f0fed9e303"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 07:52:34.855206 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.855166    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7222ab31-056f-425b-984d-27f0fed9e303-kube-api-access-g22dl" (OuterVolumeSpecName: "kube-api-access-g22dl") pod "7222ab31-056f-425b-984d-27f0fed9e303" (UID: "7222ab31-056f-425b-984d-27f0fed9e303"). InnerVolumeSpecName "kube-api-access-g22dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:52:34.954339 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.954302    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7222ab31-056f-425b-984d-27f0fed9e303-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:34.954339 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.954337    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7222ab31-056f-425b-984d-27f0fed9e303-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:34.954536 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:34.954352    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g22dl\" (UniqueName: \"kubernetes.io/projected/7222ab31-056f-425b-984d-27f0fed9e303-kube-api-access-g22dl\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:52:35.290666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:35.290577    2581 generic.go:356] "Generic (PLEG): container finished" podID="7222ab31-056f-425b-984d-27f0fed9e303" containerID="f67edd92c319ab9f5eb5769eb980b32cd25eeb6e2903ba1e2b9fb3a40f9a35a5" exitCode=0
Sep 16 07:52:35.290666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:35.290622    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"
Sep 16 07:52:35.291213 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:35.290663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm" event={"ID":"7222ab31-056f-425b-984d-27f0fed9e303","Type":"ContainerDied","Data":"f67edd92c319ab9f5eb5769eb980b32cd25eeb6e2903ba1e2b9fb3a40f9a35a5"}
Sep 16 07:52:35.291213 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:35.290712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm" event={"ID":"7222ab31-056f-425b-984d-27f0fed9e303","Type":"ContainerDied","Data":"e4b3836da34d1a4edd04048915b015f901d4216df978bc422a750d5b03664b12"}
Sep 16 07:52:35.291213 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:35.290732    2581 scope.go:117] "RemoveContainer" containerID="f67edd92c319ab9f5eb5769eb980b32cd25eeb6e2903ba1e2b9fb3a40f9a35a5"
Sep 16 07:52:35.300149 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:35.300131    2581 scope.go:117] "RemoveContainer" containerID="f67edd92c319ab9f5eb5769eb980b32cd25eeb6e2903ba1e2b9fb3a40f9a35a5"
Sep 16 07:52:35.300430 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:52:35.300410    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67edd92c319ab9f5eb5769eb980b32cd25eeb6e2903ba1e2b9fb3a40f9a35a5\": container with ID starting with f67edd92c319ab9f5eb5769eb980b32cd25eeb6e2903ba1e2b9fb3a40f9a35a5 not found: ID does not exist" containerID="f67edd92c319ab9f5eb5769eb980b32cd25eeb6e2903ba1e2b9fb3a40f9a35a5"
Sep 16 07:52:35.300475 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:35.300446    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67edd92c319ab9f5eb5769eb980b32cd25eeb6e2903ba1e2b9fb3a40f9a35a5"} err="failed to get container status \"f67edd92c319ab9f5eb5769eb980b32cd25eeb6e2903ba1e2b9fb3a40f9a35a5\": rpc error: code = NotFound desc = could not find container \"f67edd92c319ab9f5eb5769eb980b32cd25eeb6e2903ba1e2b9fb3a40f9a35a5\": container with ID starting with f67edd92c319ab9f5eb5769eb980b32cd25eeb6e2903ba1e2b9fb3a40f9a35a5 not found: ID does not exist"
Sep 16 07:52:35.315069 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:35.315039    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"]
Sep 16 07:52:35.319705 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:35.319682    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-82bbbf/explicit-model-82bbbf-kserve-66ccb4fd65-4fbtm"]
Sep 16 07:52:35.683585 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:35.683556    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7222ab31-056f-425b-984d-27f0fed9e303" path="/var/lib/kubelet/pods/7222ab31-056f-425b-984d-27f0fed9e303/volumes"
Sep 16 07:52:45.293698 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:45.293665    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"
Sep 16 07:52:54.410732 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:54.410700    2581 generic.go:356] "Generic (PLEG): container finished" podID="3418d981-b15f-4ea2-a807-75b731ea0804" containerID="795f0b061a10bd49bbf7833f8f1dc62bd052786080320164024a9e86d5600d7d" exitCode=0
Sep 16 07:52:54.411177 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:54.410755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l" event={"ID":"3418d981-b15f-4ea2-a807-75b731ea0804","Type":"ContainerDied","Data":"795f0b061a10bd49bbf7833f8f1dc62bd052786080320164024a9e86d5600d7d"}
Sep 16 07:52:54.411177 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:54.411151    2581 scope.go:117] "RemoveContainer" containerID="795f0b061a10bd49bbf7833f8f1dc62bd052786080320164024a9e86d5600d7d"
Sep 16 07:52:54.412509 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:54.412475    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:52:54.412581 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:54.412567    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:52:54.412627 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:54.412613    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:52:55.426923 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:55.425931    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l" event={"ID":"3418d981-b15f-4ea2-a807-75b731ea0804","Type":"ContainerStarted","Data":"d75fb7b051388addb56aecd0ab5267825132491bd93cf634a24f4d0b0b650989"}
Sep 16 07:52:56.295362 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:56.295319    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:56.296568 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:56.296542    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 16 07:52:56.296679 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:56.296614    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l" podUID="3418d981-b15f-4ea2-a807-75b731ea0804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 16 07:52:57.295498 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:57.295457    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 16 07:52:57.295932 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:57.295517    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l" podUID="3418d981-b15f-4ea2-a807-75b731ea0804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 16 07:52:58.249419 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:58.249385    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:52:58.296152 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:58.296117    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 16 07:52:58.296518 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:58.296173    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l" podUID="3418d981-b15f-4ea2-a807-75b731ea0804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 16 07:52:59.295509 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.295477    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 16 07:52:59.295682 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.295530    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l" podUID="3418d981-b15f-4ea2-a807-75b731ea0804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 16 07:52:59.738841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.738307    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"]
Sep 16 07:52:59.749312 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.738924    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7222ab31-056f-425b-984d-27f0fed9e303" containerName="main"
Sep 16 07:52:59.749312 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.738942    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7222ab31-056f-425b-984d-27f0fed9e303" containerName="main"
Sep 16 07:52:59.749312 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.739080    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7222ab31-056f-425b-984d-27f0fed9e303" containerName="main"
Sep 16 07:52:59.749312 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.743881    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:52:59.749312 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.746865    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-984c1a-openshift-default-dockercfg-bxs9k\""
Sep 16 07:52:59.760234 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.760206    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"]
Sep 16 07:52:59.923601 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.923563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7888826b-cd84-4cc6-b808-e18a75041202-istio-podinfo\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:52:59.923942 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.923919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7888826b-cd84-4cc6-b808-e18a75041202-istio-token\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:52:59.924162 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.924145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-istio-envoy\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:52:59.924331 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.924314    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8mbh\" (UniqueName: \"kubernetes.io/projected/7888826b-cd84-4cc6-b808-e18a75041202-kube-api-access-p8mbh\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:52:59.924486 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.924470    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-workload-socket\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:52:59.924627 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.924611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7888826b-cd84-4cc6-b808-e18a75041202-istiod-ca-cert\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:52:59.924786 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.924769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-credential-socket\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:52:59.924967 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.924951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-istio-data\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:52:59.925107 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:52:59.925092    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-workload-certs\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.028838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.026124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-workload-socket\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.028838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.026183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7888826b-cd84-4cc6-b808-e18a75041202-istiod-ca-cert\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.028838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.026228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-credential-socket\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.028838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.026265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-istio-data\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.028838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.026293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-workload-certs\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.028838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.026376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7888826b-cd84-4cc6-b808-e18a75041202-istio-podinfo\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.028838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.026401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7888826b-cd84-4cc6-b808-e18a75041202-istio-token\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.028838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.026464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-istio-envoy\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.028838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.026506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8mbh\" (UniqueName: \"kubernetes.io/projected/7888826b-cd84-4cc6-b808-e18a75041202-kube-api-access-p8mbh\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.028838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.027261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-workload-socket\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.028838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.027926    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7888826b-cd84-4cc6-b808-e18a75041202-istiod-ca-cert\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.028838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.028180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-credential-socket\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.028838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.028413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-istio-data\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.028838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.028638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-workload-certs\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.032688 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.032433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-istio-envoy\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.032688 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.032504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7888826b-cd84-4cc6-b808-e18a75041202-istio-podinfo\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.042477 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.042454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8mbh\" (UniqueName: \"kubernetes.io/projected/7888826b-cd84-4cc6-b808-e18a75041202-kube-api-access-p8mbh\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.043070 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.043048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7888826b-cd84-4cc6-b808-e18a75041202-istio-token\") pod \"e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") " pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.059875 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.059852    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:00.296301 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.296221    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 16 07:53:00.296301 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.296286    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l" podUID="3418d981-b15f-4ea2-a807-75b731ea0804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 16 07:53:00.341366 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.341246    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"]
Sep 16 07:53:00.347150 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:53:00.347121    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7888826b_cd84_4cc6_b808_e18a75041202.slice/crio-e02654bb3b737c58219c3feca4f238c67b3a36c1e034f2a6464bf633f333dd61 WatchSource:0}: Error finding container e02654bb3b737c58219c3feca4f238c67b3a36c1e034f2a6464bf633f333dd61: Status 404 returned error can't find the container with id e02654bb3b737c58219c3feca4f238c67b3a36c1e034f2a6464bf633f333dd61
Sep 16 07:53:00.349739 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.349706    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:53:00.349864 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.349792    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:53:00.349945 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.349921    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:53:00.462838 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.462774    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2" event={"ID":"7888826b-cd84-4cc6-b808-e18a75041202","Type":"ContainerStarted","Data":"839bfe8c3df14623e30bf98544b74d8fb604aae31e857f340a19f718033837de"}
Sep 16 07:53:00.463023 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.462878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2" event={"ID":"7888826b-cd84-4cc6-b808-e18a75041202","Type":"ContainerStarted","Data":"e02654bb3b737c58219c3feca4f238c67b3a36c1e034f2a6464bf633f333dd61"}
Sep 16 07:53:00.483800 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:00.483744    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2" podStartSLOduration=1.483728464 podStartE2EDuration="1.483728464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:53:00.481993921 +0000 UTC m=+1141.408606074" watchObservedRunningTime="2026-09-16 07:53:00.483728464 +0000 UTC m=+1141.410340686"
Sep 16 07:53:01.060280 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:01.060248    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:01.065281 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:01.065256    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:01.300024 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:01.299977    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:53:01.300926 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:01.300904    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 07:53:01.468615 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:01.468580    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:01.469501 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:01.469479    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:07.715629 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.715590    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"]
Sep 16 07:53:07.719423 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.719404    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"
Sep 16 07:53:07.721961 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.721941    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-984c1a\""
Sep 16 07:53:07.732078 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.732052    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"]
Sep 16 07:53:07.743385 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.743355    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"]
Sep 16 07:53:07.747926 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.747894    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"
Sep 16 07:53:07.750641 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.750609    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-984c1a-dockercfg-lspl6\""
Sep 16 07:53:07.764091 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.764062    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"]
Sep 16 07:53:07.818166 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.818133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881465a0-eb62-4314-b5d2-eb180a5d600d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n\" (UID: \"881465a0-eb62-4314-b5d2-eb180a5d600d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"
Sep 16 07:53:07.818338 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.818183    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjsjm\" (UniqueName: \"kubernetes.io/projected/881465a0-eb62-4314-b5d2-eb180a5d600d-kube-api-access-xjsjm\") pod \"payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n\" (UID: \"881465a0-eb62-4314-b5d2-eb180a5d600d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"
Sep 16 07:53:07.818338 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.818279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6glld\" (UniqueName: \"kubernetes.io/projected/41a6be09-4aaf-42c7-a00c-33cb8bcb3f45-kube-api-access-6glld\") pod \"payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx\" (UID: \"41a6be09-4aaf-42c7-a00c-33cb8bcb3f45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"
Sep 16 07:53:07.818338 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.818311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a6be09-4aaf-42c7-a00c-33cb8bcb3f45-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx\" (UID: \"41a6be09-4aaf-42c7-a00c-33cb8bcb3f45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"
Sep 16 07:53:07.919873 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.919813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881465a0-eb62-4314-b5d2-eb180a5d600d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n\" (UID: \"881465a0-eb62-4314-b5d2-eb180a5d600d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"
Sep 16 07:53:07.920047 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.919914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjsjm\" (UniqueName: \"kubernetes.io/projected/881465a0-eb62-4314-b5d2-eb180a5d600d-kube-api-access-xjsjm\") pod \"payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n\" (UID: \"881465a0-eb62-4314-b5d2-eb180a5d600d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"
Sep 16 07:53:07.920047 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.919990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6glld\" (UniqueName: \"kubernetes.io/projected/41a6be09-4aaf-42c7-a00c-33cb8bcb3f45-kube-api-access-6glld\") pod \"payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx\" (UID: \"41a6be09-4aaf-42c7-a00c-33cb8bcb3f45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"
Sep 16 07:53:07.920047 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.920026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a6be09-4aaf-42c7-a00c-33cb8bcb3f45-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx\" (UID: \"41a6be09-4aaf-42c7-a00c-33cb8bcb3f45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"
Sep 16 07:53:07.920607 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.920583    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881465a0-eb62-4314-b5d2-eb180a5d600d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n\" (UID: \"881465a0-eb62-4314-b5d2-eb180a5d600d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"
Sep 16 07:53:07.920713 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.920695    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a6be09-4aaf-42c7-a00c-33cb8bcb3f45-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx\" (UID: \"41a6be09-4aaf-42c7-a00c-33cb8bcb3f45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"
Sep 16 07:53:07.929855 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.929809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjsjm\" (UniqueName: \"kubernetes.io/projected/881465a0-eb62-4314-b5d2-eb180a5d600d-kube-api-access-xjsjm\") pod \"payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n\" (UID: \"881465a0-eb62-4314-b5d2-eb180a5d600d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"
Sep 16 07:53:07.929967 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:07.929879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6glld\" (UniqueName: \"kubernetes.io/projected/41a6be09-4aaf-42c7-a00c-33cb8bcb3f45-kube-api-access-6glld\") pod \"payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx\" (UID: \"41a6be09-4aaf-42c7-a00c-33cb8bcb3f45\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"
Sep 16 07:53:08.031573 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:08.031494    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"
Sep 16 07:53:08.062343 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:08.058749    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"
Sep 16 07:53:08.247001 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:08.246970    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"]
Sep 16 07:53:08.247218 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:08.247195    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2" podUID="0c4509ac-8880-4728-8620-85e445069160" containerName="istio-proxy" containerID="cri-o://271e425c851a557d5b29690799322073f063920ffe576ac4e0d247a6cdc7d18c" gracePeriod=30
Sep 16 07:53:08.312628 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:08.312606    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"]
Sep 16 07:53:08.314490 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:53:08.314463    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a6be09_4aaf_42c7_a00c_33cb8bcb3f45.slice/crio-2f9df9b5313af797e1845d0a675452b5e86ff1fbaad452c185c3050b1fabd007 WatchSource:0}: Error finding container 2f9df9b5313af797e1845d0a675452b5e86ff1fbaad452c185c3050b1fabd007: Status 404 returned error can't find the container with id 2f9df9b5313af797e1845d0a675452b5e86ff1fbaad452c185c3050b1fabd007
Sep 16 07:53:08.361886 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:08.359986    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"]
Sep 16 07:53:08.507804 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:08.507772    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx" event={"ID":"41a6be09-4aaf-42c7-a00c-33cb8bcb3f45","Type":"ContainerStarted","Data":"210706d31e345bdf41f7b3373a83b20b530316bcc478b138c4b2ba725c88137b"}
Sep 16 07:53:08.508000 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:08.507833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx" event={"ID":"41a6be09-4aaf-42c7-a00c-33cb8bcb3f45","Type":"ContainerStarted","Data":"2f9df9b5313af797e1845d0a675452b5e86ff1fbaad452c185c3050b1fabd007"}
Sep 16 07:53:08.508000 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:08.507982    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"
Sep 16 07:53:08.509576 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:08.509547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n" event={"ID":"881465a0-eb62-4314-b5d2-eb180a5d600d","Type":"ContainerStarted","Data":"01304911a85ccd4813ed79daac63fc513bd0882985f9da2e2bc5be4e9c1def34"}
Sep 16 07:53:08.509684 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:08.509601    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n" event={"ID":"881465a0-eb62-4314-b5d2-eb180a5d600d","Type":"ContainerStarted","Data":"96a88a6fb832b21c0956835926a6a511c6c973d6ffa086e9bf85131ee58e1a65"}
Sep 16 07:53:08.509684 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:08.509630    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"
Sep 16 07:53:08.537144 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:08.537102    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx" podStartSLOduration=1.53708995 podStartE2EDuration="1.53708995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:53:08.536716857 +0000 UTC m=+1149.463329014" watchObservedRunningTime="2026-09-16 07:53:08.53708995 +0000 UTC m=+1149.463702102"
Sep 16 07:53:08.564946 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:08.564864    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n" podStartSLOduration=1.564851488 podStartE2EDuration="1.564851488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:53:08.564240669 +0000 UTC m=+1149.490852822" watchObservedRunningTime="2026-09-16 07:53:08.564851488 +0000 UTC m=+1149.491463638"
Sep 16 07:53:13.494248 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.494221    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:53:13.541842 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.541742    2581 generic.go:356] "Generic (PLEG): container finished" podID="0c4509ac-8880-4728-8620-85e445069160" containerID="271e425c851a557d5b29690799322073f063920ffe576ac4e0d247a6cdc7d18c" exitCode=0
Sep 16 07:53:13.541842 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.541785    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"
Sep 16 07:53:13.542012 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.541811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2" event={"ID":"0c4509ac-8880-4728-8620-85e445069160","Type":"ContainerDied","Data":"271e425c851a557d5b29690799322073f063920ffe576ac4e0d247a6cdc7d18c"}
Sep 16 07:53:13.542012 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.541893    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2" event={"ID":"0c4509ac-8880-4728-8620-85e445069160","Type":"ContainerDied","Data":"4ab0719481b394c4f3592c4cae906ed073f84f37b64d40a62b846332617e1b2f"}
Sep 16 07:53:13.542012 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.541926    2581 scope.go:117] "RemoveContainer" containerID="271e425c851a557d5b29690799322073f063920ffe576ac4e0d247a6cdc7d18c"
Sep 16 07:53:13.551079 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.551061    2581 scope.go:117] "RemoveContainer" containerID="d8ff7e7327a7fd5e3d326d3258ce0183496d59bc72c0950def2af32ed25e5f6e"
Sep 16 07:53:13.559191 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.559164    2581 scope.go:117] "RemoveContainer" containerID="271e425c851a557d5b29690799322073f063920ffe576ac4e0d247a6cdc7d18c"
Sep 16 07:53:13.559446 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:53:13.559424    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271e425c851a557d5b29690799322073f063920ffe576ac4e0d247a6cdc7d18c\": container with ID starting with 271e425c851a557d5b29690799322073f063920ffe576ac4e0d247a6cdc7d18c not found: ID does not exist" containerID="271e425c851a557d5b29690799322073f063920ffe576ac4e0d247a6cdc7d18c"
Sep 16 07:53:13.559556 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.559451    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271e425c851a557d5b29690799322073f063920ffe576ac4e0d247a6cdc7d18c"} err="failed to get container status \"271e425c851a557d5b29690799322073f063920ffe576ac4e0d247a6cdc7d18c\": rpc error: code = NotFound desc = could not find container \"271e425c851a557d5b29690799322073f063920ffe576ac4e0d247a6cdc7d18c\": container with ID starting with 271e425c851a557d5b29690799322073f063920ffe576ac4e0d247a6cdc7d18c not found: ID does not exist"
Sep 16 07:53:13.559556 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.559468    2581 scope.go:117] "RemoveContainer" containerID="d8ff7e7327a7fd5e3d326d3258ce0183496d59bc72c0950def2af32ed25e5f6e"
Sep 16 07:53:13.559713 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:53:13.559697    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ff7e7327a7fd5e3d326d3258ce0183496d59bc72c0950def2af32ed25e5f6e\": container with ID starting with d8ff7e7327a7fd5e3d326d3258ce0183496d59bc72c0950def2af32ed25e5f6e not found: ID does not exist" containerID="d8ff7e7327a7fd5e3d326d3258ce0183496d59bc72c0950def2af32ed25e5f6e"
Sep 16 07:53:13.559749 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.559718    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ff7e7327a7fd5e3d326d3258ce0183496d59bc72c0950def2af32ed25e5f6e"} err="failed to get container status \"d8ff7e7327a7fd5e3d326d3258ce0183496d59bc72c0950def2af32ed25e5f6e\": rpc error: code = NotFound desc = could not find container \"d8ff7e7327a7fd5e3d326d3258ce0183496d59bc72c0950def2af32ed25e5f6e\": container with ID starting with d8ff7e7327a7fd5e3d326d3258ce0183496d59bc72c0950def2af32ed25e5f6e not found: ID does not exist"
Sep 16 07:53:13.587537 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.587517    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4509ac-8880-4728-8620-85e445069160-istio-podinfo\") pod \"0c4509ac-8880-4728-8620-85e445069160\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") "
Sep 16 07:53:13.587640 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.587550    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-workload-certs\") pod \"0c4509ac-8880-4728-8620-85e445069160\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") "
Sep 16 07:53:13.587640 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.587572    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4509ac-8880-4728-8620-85e445069160-istiod-ca-cert\") pod \"0c4509ac-8880-4728-8620-85e445069160\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") "
Sep 16 07:53:13.587640 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.587602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-istio-data\") pod \"0c4509ac-8880-4728-8620-85e445069160\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") "
Sep 16 07:53:13.587752 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.587658    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-workload-socket\") pod \"0c4509ac-8880-4728-8620-85e445069160\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") "
Sep 16 07:53:13.587752 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.587680    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4509ac-8880-4728-8620-85e445069160-istio-token\") pod \"0c4509ac-8880-4728-8620-85e445069160\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") "
Sep 16 07:53:13.587752 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.587709    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqlqz\" (UniqueName: \"kubernetes.io/projected/0c4509ac-8880-4728-8620-85e445069160-kube-api-access-wqlqz\") pod \"0c4509ac-8880-4728-8620-85e445069160\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") "
Sep 16 07:53:13.587900 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.587774    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-credential-socket\") pod \"0c4509ac-8880-4728-8620-85e445069160\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") "
Sep 16 07:53:13.587900 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.587805    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0c4509ac-8880-4728-8620-85e445069160" (UID: "0c4509ac-8880-4728-8620-85e445069160"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:53:13.587900 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.587811    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-istio-envoy\") pod \"0c4509ac-8880-4728-8620-85e445069160\" (UID: \"0c4509ac-8880-4728-8620-85e445069160\") "
Sep 16 07:53:13.588052 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.587980    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-istio-data" (OuterVolumeSpecName: "istio-data") pod "0c4509ac-8880-4728-8620-85e445069160" (UID: "0c4509ac-8880-4728-8620-85e445069160"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:53:13.588052 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.587980    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4509ac-8880-4728-8620-85e445069160-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0c4509ac-8880-4728-8620-85e445069160" (UID: "0c4509ac-8880-4728-8620-85e445069160"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:53:13.588262 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.588234    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0c4509ac-8880-4728-8620-85e445069160" (UID: "0c4509ac-8880-4728-8620-85e445069160"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:53:13.588373 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.588323    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:53:13.588373 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.588343    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4509ac-8880-4728-8620-85e445069160-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:53:13.588373 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.588355    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:53:13.588529 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.588459    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0c4509ac-8880-4728-8620-85e445069160" (UID: "0c4509ac-8880-4728-8620-85e445069160"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:53:13.589778 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.589751    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0c4509ac-8880-4728-8620-85e445069160" (UID: "0c4509ac-8880-4728-8620-85e445069160"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:53:13.589881 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.589858    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4509ac-8880-4728-8620-85e445069160-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c4509ac-8880-4728-8620-85e445069160" (UID: "0c4509ac-8880-4728-8620-85e445069160"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:53:13.590051 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.590034    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c4509ac-8880-4728-8620-85e445069160-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0c4509ac-8880-4728-8620-85e445069160" (UID: "0c4509ac-8880-4728-8620-85e445069160"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 07:53:13.590129 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.590102    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4509ac-8880-4728-8620-85e445069160-kube-api-access-wqlqz" (OuterVolumeSpecName: "kube-api-access-wqlqz") pod "0c4509ac-8880-4728-8620-85e445069160" (UID: "0c4509ac-8880-4728-8620-85e445069160"). InnerVolumeSpecName "kube-api-access-wqlqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:53:13.689405 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.689383    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4509ac-8880-4728-8620-85e445069160-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:53:13.689405 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.689404    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:53:13.689529 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.689413    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4509ac-8880-4728-8620-85e445069160-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:53:13.689529 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.689422    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqlqz\" (UniqueName: \"kubernetes.io/projected/0c4509ac-8880-4728-8620-85e445069160-kube-api-access-wqlqz\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:53:13.689529 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.689431    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:53:13.689529 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.689438    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4509ac-8880-4728-8620-85e445069160-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:53:13.859013 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.858950    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"]
Sep 16 07:53:13.864646 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:13.864622    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-82bbbf-openshift-default-5c8bb86f6-wz5c2"]
Sep 16 07:53:15.690483 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:15.690450    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c4509ac-8880-4728-8620-85e445069160" path="/var/lib/kubelet/pods/0c4509ac-8880-4728-8620-85e445069160/volumes"
Sep 16 07:53:19.518092 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:19.518040    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"
Sep 16 07:53:19.518574 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:19.518514    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"
Sep 16 07:53:25.757315 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.757275    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"]
Sep 16 07:53:25.757892 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.757872    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c4509ac-8880-4728-8620-85e445069160" containerName="istio-proxy"
Sep 16 07:53:25.757986 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.757893    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c4509ac-8880-4728-8620-85e445069160" containerName="istio-proxy"
Sep 16 07:53:25.757986 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.757913    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c4509ac-8880-4728-8620-85e445069160" containerName="istio-proxy"
Sep 16 07:53:25.757986 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.757922    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c4509ac-8880-4728-8620-85e445069160" containerName="istio-proxy"
Sep 16 07:53:25.758140 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.758073    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c4509ac-8880-4728-8620-85e445069160" containerName="istio-proxy"
Sep 16 07:53:25.758140 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.758087    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c4509ac-8880-4728-8620-85e445069160" containerName="istio-proxy"
Sep 16 07:53:25.762528 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.762508    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:25.766198 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.766170    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"default-dockercfg-dkpl6\""
Sep 16 07:53:25.766314 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.766175    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"kube-root-ca.crt\""
Sep 16 07:53:25.766314 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.766181    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"e2e-conflict-a-8a3fd4d9-kserve-self-signed-certs\""
Sep 16 07:53:25.766314 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.766181    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"openshift-service-ca.crt\""
Sep 16 07:53:25.770089 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.770072    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"]
Sep 16 07:53:25.910625 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.910588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-tls-certs\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:25.910800 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.910644    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-home\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:25.910800 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.910732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-dshm\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:25.910976 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.910811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-tmp-dir\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:25.910976 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.910868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-model-cache\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:25.910976 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:25.910908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmtc6\" (UniqueName: \"kubernetes.io/projected/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-kube-api-access-zmtc6\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:26.012649 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.012577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-tls-certs\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:26.012770 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.012652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-home\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:26.012770 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.012683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-dshm\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:26.012906 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.012841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-tmp-dir\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:26.012906 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.012894    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-model-cache\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:26.013014 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.012927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmtc6\" (UniqueName: \"kubernetes.io/projected/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-kube-api-access-zmtc6\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:26.013224 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.013197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-home\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:26.013315 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.013262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-model-cache\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:26.013438 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.013421    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-tmp-dir\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:26.015313 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.015279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-tls-certs\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:26.015441 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.015419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-dshm\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:26.024342 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.024317    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmtc6\" (UniqueName: \"kubernetes.io/projected/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-kube-api-access-zmtc6\") pod \"e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:26.073855 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.073804    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:26.350140 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.350037    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"]
Sep 16 07:53:26.354424 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:53:26.354398    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92bf24cf_12a3_4f8c_b6e5_139a91e77a25.slice/crio-de1fb65c365915aedb9c6dd975ba31d417c8ac647c7654d2784f67ee40fa959b WatchSource:0}: Error finding container de1fb65c365915aedb9c6dd975ba31d417c8ac647c7654d2784f67ee40fa959b: Status 404 returned error can't find the container with id de1fb65c365915aedb9c6dd975ba31d417c8ac647c7654d2784f67ee40fa959b
Sep 16 07:53:26.617717 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.617625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn" event={"ID":"92bf24cf-12a3-4f8c-b6e5-139a91e77a25","Type":"ContainerStarted","Data":"5b58cf4bcdd150db64800baf9d69ddcc71ad7fabfd33d20bc73aa34ca5f681f1"}
Sep 16 07:53:26.617717 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.617668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn" event={"ID":"92bf24cf-12a3-4f8c-b6e5-139a91e77a25","Type":"ContainerStarted","Data":"de1fb65c365915aedb9c6dd975ba31d417c8ac647c7654d2784f67ee40fa959b"}
Sep 16 07:53:26.637656 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:26.637607    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn" podStartSLOduration=1.637592843 podStartE2EDuration="1.637592843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:53:26.635606091 +0000 UTC m=+1167.562218244" watchObservedRunningTime="2026-09-16 07:53:26.637592843 +0000 UTC m=+1167.564204995"
Sep 16 07:53:27.625392 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:27.625356    2581 generic.go:356] "Generic (PLEG): container finished" podID="7888826b-cd84-4cc6-b808-e18a75041202" containerID="839bfe8c3df14623e30bf98544b74d8fb604aae31e857f340a19f718033837de" exitCode=0
Sep 16 07:53:27.625793 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:27.625432    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2" event={"ID":"7888826b-cd84-4cc6-b808-e18a75041202","Type":"ContainerDied","Data":"839bfe8c3df14623e30bf98544b74d8fb604aae31e857f340a19f718033837de"}
Sep 16 07:53:27.625961 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:27.625946    2581 scope.go:117] "RemoveContainer" containerID="839bfe8c3df14623e30bf98544b74d8fb604aae31e857f340a19f718033837de"
Sep 16 07:53:27.627138 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:27.627111    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:53:27.627210 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:27.627177    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:53:27.627250 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:27.627211    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:53:28.637075 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:28.637033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2" event={"ID":"7888826b-cd84-4cc6-b808-e18a75041202","Type":"ContainerStarted","Data":"4d564fa0976c0b37912f2c83d28e27b0179d5a8b02092900ee5fe0fb4a196640"}
Sep 16 07:53:29.060372 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:29.060337    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:29.061410 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:29.061381    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 07:53:29.061532 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:29.061445    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2" podUID="7888826b-cd84-4cc6-b808-e18a75041202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 07:53:30.060525 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:30.060494    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 07:53:30.060895 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:30.060549    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2" podUID="7888826b-cd84-4cc6-b808-e18a75041202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 07:53:31.060410 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:31.060373    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 07:53:31.060588 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:31.060427    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2" podUID="7888826b-cd84-4cc6-b808-e18a75041202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 07:53:31.468516 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:31.468481    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:32.060232 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:32.060197    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 07:53:32.060407 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:32.060252    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2" podUID="7888826b-cd84-4cc6-b808-e18a75041202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 07:53:33.061018 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:33.060982    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 07:53:33.061388 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:33.061035    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2" podUID="7888826b-cd84-4cc6-b808-e18a75041202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 07:53:34.064456 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:34.064423    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:34.065443 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:34.065423    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 07:53:36.074659 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:36.074628    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:36.075061 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:36.074682    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:36.087296 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:36.087273    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:36.693097 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:36.693068    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:53:41.789982 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.789954    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"]
Sep 16 07:53:41.796843 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.794360    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.800692 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.800666    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fc2a294d-openshift-default-dockercfg-7wv2d\""
Sep 16 07:53:41.806420 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.806394    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"]
Sep 16 07:53:41.870982 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.870951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-data\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.870982 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.870982    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-podinfo\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.871162 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.871026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt448\" (UniqueName: \"kubernetes.io/projected/b091dc61-ca45-4d41-8c11-4b08d23eff42-kube-api-access-tt448\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.871162 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.871109    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-credential-socket\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.871162 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.871149    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-envoy\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.871495 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.871165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b091dc61-ca45-4d41-8c11-4b08d23eff42-istiod-ca-cert\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.871495 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.871241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-workload-certs\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.871495 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.871296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-workload-socket\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.871495 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.871317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-token\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.973036 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.972998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-workload-certs\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.973674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.973647    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-workload-certs\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.973848 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.973693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-workload-socket\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.973985 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.973969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-token\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.974162 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.974146    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-data\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.974271 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.974256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-podinfo\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.974374 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.974054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-workload-socket\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.974499 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.974482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt448\" (UniqueName: \"kubernetes.io/projected/b091dc61-ca45-4d41-8c11-4b08d23eff42-kube-api-access-tt448\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.974629 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.974615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-credential-socket\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.974763 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.974749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-envoy\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.974887 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.974872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b091dc61-ca45-4d41-8c11-4b08d23eff42-istiod-ca-cert\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.975708 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.975685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b091dc61-ca45-4d41-8c11-4b08d23eff42-istiod-ca-cert\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.976459 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.976049    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-data\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.976592 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.976404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-credential-socket\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.977604 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.977580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-podinfo\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.979302 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.979282    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-envoy\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.985513 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.985471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-token\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:41.986178 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:41.986140    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt448\" (UniqueName: \"kubernetes.io/projected/b091dc61-ca45-4d41-8c11-4b08d23eff42-kube-api-access-tt448\") pod \"e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") " pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:42.112188 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:42.112085    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:42.384450 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:42.384426    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"]
Sep 16 07:53:42.386340 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:53:42.386314    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb091dc61_ca45_4d41_8c11_4b08d23eff42.slice/crio-cf84687738a2232d987c83c67090c17cbd507975461b072b4be1eb38e14c1521 WatchSource:0}: Error finding container cf84687738a2232d987c83c67090c17cbd507975461b072b4be1eb38e14c1521: Status 404 returned error can't find the container with id cf84687738a2232d987c83c67090c17cbd507975461b072b4be1eb38e14c1521
Sep 16 07:53:42.388805 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:42.388762    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:53:42.388912 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:42.388895    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:53:42.388965 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:42.388948    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:53:42.716969 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:42.716932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6" event={"ID":"b091dc61-ca45-4d41-8c11-4b08d23eff42","Type":"ContainerStarted","Data":"8e9e949f6af38dbea7a35f2d7d610529a0f28fcde5e4d35da81a099e784ce828"}
Sep 16 07:53:42.717169 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:42.716991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6" event={"ID":"b091dc61-ca45-4d41-8c11-4b08d23eff42","Type":"ContainerStarted","Data":"cf84687738a2232d987c83c67090c17cbd507975461b072b4be1eb38e14c1521"}
Sep 16 07:53:42.741192 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:42.741145    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6" podStartSLOduration=1.741129411 podStartE2EDuration="1.741129411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:53:42.739295922 +0000 UTC m=+1183.665908075" watchObservedRunningTime="2026-09-16 07:53:42.741129411 +0000 UTC m=+1183.667741564"
Sep 16 07:53:43.112875 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:43.112775    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:43.117714 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:43.117687    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:43.722145 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:43.722119    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:43.722998 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:43.722981    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:53:49.269700 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:49.269667    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"]
Sep 16 07:53:49.272081 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:49.272063    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"
Sep 16 07:53:49.274774 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:49.274751    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fc2a294d\""
Sep 16 07:53:49.274920 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:49.274847    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fc2a294d-dockercfg-ws8s7\""
Sep 16 07:53:49.282289 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:49.282268    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"]
Sep 16 07:53:49.351457 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:49.351415    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9671b553-9409-4397-9cfb-7cf62cf76515-plugins-config-volume\") pod \"payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt\" (UID: \"9671b553-9409-4397-9cfb-7cf62cf76515\") " pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"
Sep 16 07:53:49.351755 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:49.351730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrc9p\" (UniqueName: \"kubernetes.io/projected/9671b553-9409-4397-9cfb-7cf62cf76515-kube-api-access-wrc9p\") pod \"payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt\" (UID: \"9671b553-9409-4397-9cfb-7cf62cf76515\") " pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"
Sep 16 07:53:49.452674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:49.452641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9671b553-9409-4397-9cfb-7cf62cf76515-plugins-config-volume\") pod \"payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt\" (UID: \"9671b553-9409-4397-9cfb-7cf62cf76515\") " pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"
Sep 16 07:53:49.452674 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:49.452674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrc9p\" (UniqueName: \"kubernetes.io/projected/9671b553-9409-4397-9cfb-7cf62cf76515-kube-api-access-wrc9p\") pod \"payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt\" (UID: \"9671b553-9409-4397-9cfb-7cf62cf76515\") " pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"
Sep 16 07:53:49.453351 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:49.453330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9671b553-9409-4397-9cfb-7cf62cf76515-plugins-config-volume\") pod \"payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt\" (UID: \"9671b553-9409-4397-9cfb-7cf62cf76515\") " pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"
Sep 16 07:53:49.460221 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:49.460199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrc9p\" (UniqueName: \"kubernetes.io/projected/9671b553-9409-4397-9cfb-7cf62cf76515-kube-api-access-wrc9p\") pod \"payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt\" (UID: \"9671b553-9409-4397-9cfb-7cf62cf76515\") " pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"
Sep 16 07:53:49.583585 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:49.583518    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"
Sep 16 07:53:49.854077 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:49.854045    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"]
Sep 16 07:53:49.855409 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:53:49.855384    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9671b553_9409_4397_9cfb_7cf62cf76515.slice/crio-d144ecb28dd5ba12358d943e71422822e5849bc1ba79119b82904e987d8691f2 WatchSource:0}: Error finding container d144ecb28dd5ba12358d943e71422822e5849bc1ba79119b82904e987d8691f2: Status 404 returned error can't find the container with id d144ecb28dd5ba12358d943e71422822e5849bc1ba79119b82904e987d8691f2
Sep 16 07:53:50.765026 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:50.764994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt" event={"ID":"9671b553-9409-4397-9cfb-7cf62cf76515","Type":"ContainerStarted","Data":"0004b44f4ee1c454aa15741d2aeb17dc987d62f7d1f8302dd75ccc918f7db6fb"}
Sep 16 07:53:50.765415 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:50.765047    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt" event={"ID":"9671b553-9409-4397-9cfb-7cf62cf76515","Type":"ContainerStarted","Data":"d144ecb28dd5ba12358d943e71422822e5849bc1ba79119b82904e987d8691f2"}
Sep 16 07:53:50.765415 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:50.765068    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"
Sep 16 07:53:50.783830 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:50.783759    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt" podStartSLOduration=1.783740796 podStartE2EDuration="1.783740796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:53:50.783050698 +0000 UTC m=+1191.709662851" watchObservedRunningTime="2026-09-16 07:53:50.783740796 +0000 UTC m=+1191.710352950"
Sep 16 07:53:59.595593 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:59.595564    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 07:53:59.597725 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:53:59.597699    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 07:54:01.774367 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:01.774331    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"
Sep 16 07:54:07.877134 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:07.877098    2581 generic.go:356] "Generic (PLEG): container finished" podID="b091dc61-ca45-4d41-8c11-4b08d23eff42" containerID="8e9e949f6af38dbea7a35f2d7d610529a0f28fcde5e4d35da81a099e784ce828" exitCode=0
Sep 16 07:54:07.877558 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:07.877170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6" event={"ID":"b091dc61-ca45-4d41-8c11-4b08d23eff42","Type":"ContainerDied","Data":"8e9e949f6af38dbea7a35f2d7d610529a0f28fcde5e4d35da81a099e784ce828"}
Sep 16 07:54:07.877620 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:07.877605    2581 scope.go:117] "RemoveContainer" containerID="8e9e949f6af38dbea7a35f2d7d610529a0f28fcde5e4d35da81a099e784ce828"
Sep 16 07:54:07.878965 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:07.878939    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:54:07.879028 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:07.879006    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:54:07.879065 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:07.879039    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:54:08.884807 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:08.884771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6" event={"ID":"b091dc61-ca45-4d41-8c11-4b08d23eff42","Type":"ContainerStarted","Data":"0232567d564ba7a5505aace1af649b5203a45cc15e8bd4dfddd8efaa172ab13c"}
Sep 16 07:54:09.112915 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:09.112876    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:54:09.114479 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:09.114158    2581 patch_prober.go:28] interesting pod/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 16 07:54:09.114479 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:09.114216    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6" podUID="b091dc61-ca45-4d41-8c11-4b08d23eff42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 07:54:10.112557 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:10.112523    2581 patch_prober.go:28] interesting pod/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 16 07:54:10.113142 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:10.113109    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6" podUID="b091dc61-ca45-4d41-8c11-4b08d23eff42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 07:54:11.113131 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:11.113095    2581 patch_prober.go:28] interesting pod/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 16 07:54:11.113722 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:11.113689    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6" podUID="b091dc61-ca45-4d41-8c11-4b08d23eff42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 07:54:12.124298 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:12.124257    2581 patch_prober.go:28] interesting pod/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 16 07:54:12.124916 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:12.124881    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6" podUID="b091dc61-ca45-4d41-8c11-4b08d23eff42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 07:54:13.112941 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:13.112899    2581 patch_prober.go:28] interesting pod/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 16 07:54:13.113149 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:13.112964    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6" podUID="b091dc61-ca45-4d41-8c11-4b08d23eff42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 16 07:54:13.722929 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:13.722894    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:54:14.129366 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:14.129279    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:54:14.130742 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:14.130718    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:54:22.881418 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:22.881384    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"]
Sep 16 07:54:22.881907 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:22.881597    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8" podUID="ac0b949d-481f-4dba-8da8-08adfabc40ee" containerName="payload-processing" containerID="cri-o://4960207c8f6c77c4f83f7138daaf89e3add9700a4f3df837cac0e3f5f028fc94" gracePeriod=30
Sep 16 07:54:22.892451 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:22.892410    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"]
Sep 16 07:54:22.892999 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:22.892670    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv" podUID="d5e9831a-5cea-44bf-8e19-0670680f33c8" containerName="payload-pre-processing" containerID="cri-o://d7c7cffc5a8aa5ae83ee5d67923ee7047c2b4f5dd6a410eff33eef5e12f91940" gracePeriod=30
Sep 16 07:54:23.144766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.144708    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"
Sep 16 07:54:23.194569 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.194545    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"
Sep 16 07:54:23.290465 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.290434    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0b949d-481f-4dba-8da8-08adfabc40ee-plugins-config-volume\") pod \"ac0b949d-481f-4dba-8da8-08adfabc40ee\" (UID: \"ac0b949d-481f-4dba-8da8-08adfabc40ee\") "
Sep 16 07:54:23.290639 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.290531    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e9831a-5cea-44bf-8e19-0670680f33c8-plugins-config-volume\") pod \"d5e9831a-5cea-44bf-8e19-0670680f33c8\" (UID: \"d5e9831a-5cea-44bf-8e19-0670680f33c8\") "
Sep 16 07:54:23.290754 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.290720    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m69qd\" (UniqueName: \"kubernetes.io/projected/d5e9831a-5cea-44bf-8e19-0670680f33c8-kube-api-access-m69qd\") pod \"d5e9831a-5cea-44bf-8e19-0670680f33c8\" (UID: \"d5e9831a-5cea-44bf-8e19-0670680f33c8\") "
Sep 16 07:54:23.290911 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.290796    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5l96\" (UniqueName: \"kubernetes.io/projected/ac0b949d-481f-4dba-8da8-08adfabc40ee-kube-api-access-r5l96\") pod \"ac0b949d-481f-4dba-8da8-08adfabc40ee\" (UID: \"ac0b949d-481f-4dba-8da8-08adfabc40ee\") "
Sep 16 07:54:23.290911 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.290865    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0b949d-481f-4dba-8da8-08adfabc40ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac0b949d-481f-4dba-8da8-08adfabc40ee" (UID: "ac0b949d-481f-4dba-8da8-08adfabc40ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:54:23.290911 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.290895    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e9831a-5cea-44bf-8e19-0670680f33c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5e9831a-5cea-44bf-8e19-0670680f33c8" (UID: "d5e9831a-5cea-44bf-8e19-0670680f33c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:54:23.291186 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.291165    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0b949d-481f-4dba-8da8-08adfabc40ee-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:54:23.291186 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.291185    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5e9831a-5cea-44bf-8e19-0670680f33c8-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:54:23.292807 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.292781    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e9831a-5cea-44bf-8e19-0670680f33c8-kube-api-access-m69qd" (OuterVolumeSpecName: "kube-api-access-m69qd") pod "d5e9831a-5cea-44bf-8e19-0670680f33c8" (UID: "d5e9831a-5cea-44bf-8e19-0670680f33c8"). InnerVolumeSpecName "kube-api-access-m69qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:54:23.292929 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.292802    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0b949d-481f-4dba-8da8-08adfabc40ee-kube-api-access-r5l96" (OuterVolumeSpecName: "kube-api-access-r5l96") pod "ac0b949d-481f-4dba-8da8-08adfabc40ee" (UID: "ac0b949d-481f-4dba-8da8-08adfabc40ee"). InnerVolumeSpecName "kube-api-access-r5l96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:54:23.392052 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.392017    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m69qd\" (UniqueName: \"kubernetes.io/projected/d5e9831a-5cea-44bf-8e19-0670680f33c8-kube-api-access-m69qd\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:54:23.392052 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.392048    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5l96\" (UniqueName: \"kubernetes.io/projected/ac0b949d-481f-4dba-8da8-08adfabc40ee-kube-api-access-r5l96\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:54:23.973650 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.973609    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5e9831a-5cea-44bf-8e19-0670680f33c8" containerID="d7c7cffc5a8aa5ae83ee5d67923ee7047c2b4f5dd6a410eff33eef5e12f91940" exitCode=0
Sep 16 07:54:23.974054 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.973652    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"
Sep 16 07:54:23.974054 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.973698    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv" event={"ID":"d5e9831a-5cea-44bf-8e19-0670680f33c8","Type":"ContainerDied","Data":"d7c7cffc5a8aa5ae83ee5d67923ee7047c2b4f5dd6a410eff33eef5e12f91940"}
Sep 16 07:54:23.974054 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.973748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv" event={"ID":"d5e9831a-5cea-44bf-8e19-0670680f33c8","Type":"ContainerDied","Data":"c0a5403785d143a3f2ce903905d9dbf7533500691822fab4946ddfc6b55c8bdc"}
Sep 16 07:54:23.974054 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.973770    2581 scope.go:117] "RemoveContainer" containerID="d7c7cffc5a8aa5ae83ee5d67923ee7047c2b4f5dd6a410eff33eef5e12f91940"
Sep 16 07:54:23.975311 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.975291    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac0b949d-481f-4dba-8da8-08adfabc40ee" containerID="4960207c8f6c77c4f83f7138daaf89e3add9700a4f3df837cac0e3f5f028fc94" exitCode=0
Sep 16 07:54:23.975420 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.975334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8" event={"ID":"ac0b949d-481f-4dba-8da8-08adfabc40ee","Type":"ContainerDied","Data":"4960207c8f6c77c4f83f7138daaf89e3add9700a4f3df837cac0e3f5f028fc94"}
Sep 16 07:54:23.975420 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.975341    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"
Sep 16 07:54:23.975420 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.975376    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8" event={"ID":"ac0b949d-481f-4dba-8da8-08adfabc40ee","Type":"ContainerDied","Data":"fea7a320edb593f09b1f3eaac3d692cb9fbaeb1983c38945fe69a03163d08599"}
Sep 16 07:54:23.982850 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.982833    2581 scope.go:117] "RemoveContainer" containerID="d7c7cffc5a8aa5ae83ee5d67923ee7047c2b4f5dd6a410eff33eef5e12f91940"
Sep 16 07:54:23.983113 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:54:23.983096    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c7cffc5a8aa5ae83ee5d67923ee7047c2b4f5dd6a410eff33eef5e12f91940\": container with ID starting with d7c7cffc5a8aa5ae83ee5d67923ee7047c2b4f5dd6a410eff33eef5e12f91940 not found: ID does not exist" containerID="d7c7cffc5a8aa5ae83ee5d67923ee7047c2b4f5dd6a410eff33eef5e12f91940"
Sep 16 07:54:23.983174 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.983118    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c7cffc5a8aa5ae83ee5d67923ee7047c2b4f5dd6a410eff33eef5e12f91940"} err="failed to get container status \"d7c7cffc5a8aa5ae83ee5d67923ee7047c2b4f5dd6a410eff33eef5e12f91940\": rpc error: code = NotFound desc = could not find container \"d7c7cffc5a8aa5ae83ee5d67923ee7047c2b4f5dd6a410eff33eef5e12f91940\": container with ID starting with d7c7cffc5a8aa5ae83ee5d67923ee7047c2b4f5dd6a410eff33eef5e12f91940 not found: ID does not exist"
Sep 16 07:54:23.983174 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.983132    2581 scope.go:117] "RemoveContainer" containerID="4960207c8f6c77c4f83f7138daaf89e3add9700a4f3df837cac0e3f5f028fc94"
Sep 16 07:54:23.991380 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.991350    2581 scope.go:117] "RemoveContainer" containerID="4960207c8f6c77c4f83f7138daaf89e3add9700a4f3df837cac0e3f5f028fc94"
Sep 16 07:54:23.991644 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:54:23.991621    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4960207c8f6c77c4f83f7138daaf89e3add9700a4f3df837cac0e3f5f028fc94\": container with ID starting with 4960207c8f6c77c4f83f7138daaf89e3add9700a4f3df837cac0e3f5f028fc94 not found: ID does not exist" containerID="4960207c8f6c77c4f83f7138daaf89e3add9700a4f3df837cac0e3f5f028fc94"
Sep 16 07:54:23.991711 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.991653    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4960207c8f6c77c4f83f7138daaf89e3add9700a4f3df837cac0e3f5f028fc94"} err="failed to get container status \"4960207c8f6c77c4f83f7138daaf89e3add9700a4f3df837cac0e3f5f028fc94\": rpc error: code = NotFound desc = could not find container \"4960207c8f6c77c4f83f7138daaf89e3add9700a4f3df837cac0e3f5f028fc94\": container with ID starting with 4960207c8f6c77c4f83f7138daaf89e3add9700a4f3df837cac0e3f5f028fc94 not found: ID does not exist"
Sep 16 07:54:23.993098 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.993078    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"]
Sep 16 07:54:23.999530 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:23.999507    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5dd30f-7bd77ff94b-stbbv"]
Sep 16 07:54:24.009875 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:24.009854    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"]
Sep 16 07:54:24.013975 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:24.013952    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5dd30f-bb4f64b57-vdwk8"]
Sep 16 07:54:25.685092 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:25.685058    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac0b949d-481f-4dba-8da8-08adfabc40ee" path="/var/lib/kubelet/pods/ac0b949d-481f-4dba-8da8-08adfabc40ee/volumes"
Sep 16 07:54:25.685447 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:25.685407    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5e9831a-5cea-44bf-8e19-0670680f33c8" path="/var/lib/kubelet/pods/d5e9831a-5cea-44bf-8e19-0670680f33c8/volumes"
Sep 16 07:54:34.434607 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:34.434570    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"]
Sep 16 07:54:34.435485 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:34.435451    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp" podUID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerName="istio-proxy" containerID="cri-o://1f8b616d3dea9802949515cdd5a1baec743317cf2dbfac3368ce54c9f25a2fce" gracePeriod=30
Sep 16 07:54:34.793032 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:34.792939    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn"]
Sep 16 07:54:34.793443 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:34.793424    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5e9831a-5cea-44bf-8e19-0670680f33c8" containerName="payload-pre-processing"
Sep 16 07:54:34.793543 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:34.793446    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5e9831a-5cea-44bf-8e19-0670680f33c8" containerName="payload-pre-processing"
Sep 16 07:54:34.793543 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:34.793477    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac0b949d-481f-4dba-8da8-08adfabc40ee" containerName="payload-processing"
Sep 16 07:54:34.793543 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:34.793485    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac0b949d-481f-4dba-8da8-08adfabc40ee" containerName="payload-processing"
Sep 16 07:54:34.793695 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:34.793597    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac0b949d-481f-4dba-8da8-08adfabc40ee" containerName="payload-processing"
Sep 16 07:54:34.793695 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:34.793612    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5e9831a-5cea-44bf-8e19-0670680f33c8" containerName="payload-pre-processing"
Sep 16 07:54:34.796936 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:34.796916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn"
Sep 16 07:54:34.799331 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:34.799308    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\""
Sep 16 07:54:34.799421 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:34.799315    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 07:54:34.806302 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:34.806279    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn"]
Sep 16 07:54:34.900647 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:34.900623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a7acdd-045d-408d-81e9-5003dd696975-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn\" (UID: \"94a7acdd-045d-408d-81e9-5003dd696975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn"
Sep 16 07:54:35.001925 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:35.001897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a7acdd-045d-408d-81e9-5003dd696975-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn\" (UID: \"94a7acdd-045d-408d-81e9-5003dd696975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn"
Sep 16 07:54:35.002450 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:35.002431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a7acdd-045d-408d-81e9-5003dd696975-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn\" (UID: \"94a7acdd-045d-408d-81e9-5003dd696975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn"
Sep 16 07:54:35.108676 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:35.108614    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn"
Sep 16 07:54:35.378975 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:35.378949    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn"]
Sep 16 07:54:35.384140 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:54:35.384108    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a7acdd_045d_408d_81e9_5003dd696975.slice/crio-eb2027dbdd05919d60eceeddc7e869fbb930363ff5b2ef1cc024634728ff5f17 WatchSource:0}: Error finding container eb2027dbdd05919d60eceeddc7e869fbb930363ff5b2ef1cc024634728ff5f17: Status 404 returned error can't find the container with id eb2027dbdd05919d60eceeddc7e869fbb930363ff5b2ef1cc024634728ff5f17
Sep 16 07:54:36.052301 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:36.052260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn" event={"ID":"94a7acdd-045d-408d-81e9-5003dd696975","Type":"ContainerStarted","Data":"528f7b5aa0b2f6acdf944d8eebd773ba5f3e46de58a15dce0a7320bc9a1d9d0e"}
Sep 16 07:54:36.052684 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:36.052355    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn" event={"ID":"94a7acdd-045d-408d-81e9-5003dd696975","Type":"ContainerStarted","Data":"eb2027dbdd05919d60eceeddc7e869fbb930363ff5b2ef1cc024634728ff5f17"}
Sep 16 07:54:36.068435 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:36.068390    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn" podStartSLOduration=2.06837637 podStartE2EDuration="2.06837637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:54:36.065806159 +0000 UTC m=+1236.992418310" watchObservedRunningTime="2026-09-16 07:54:36.06837637 +0000 UTC m=+1236.994988520"
Sep 16 07:54:37.058965 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:37.058926    2581 generic.go:356] "Generic (PLEG): container finished" podID="94a7acdd-045d-408d-81e9-5003dd696975" containerID="528f7b5aa0b2f6acdf944d8eebd773ba5f3e46de58a15dce0a7320bc9a1d9d0e" exitCode=0
Sep 16 07:54:37.059348 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:37.058972    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn" event={"ID":"94a7acdd-045d-408d-81e9-5003dd696975","Type":"ContainerDied","Data":"528f7b5aa0b2f6acdf944d8eebd773ba5f3e46de58a15dce0a7320bc9a1d9d0e"}
Sep 16 07:54:38.201752 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:38.201723    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn"
Sep 16 07:54:38.335417 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:38.335334    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a7acdd-045d-408d-81e9-5003dd696975-maas-api-service-ca\") pod \"94a7acdd-045d-408d-81e9-5003dd696975\" (UID: \"94a7acdd-045d-408d-81e9-5003dd696975\") "
Sep 16 07:54:38.335667 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:38.335645    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a7acdd-045d-408d-81e9-5003dd696975-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94a7acdd-045d-408d-81e9-5003dd696975" (UID: "94a7acdd-045d-408d-81e9-5003dd696975"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:54:38.436938 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:38.436906    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a7acdd-045d-408d-81e9-5003dd696975-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:54:39.070294 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.070263    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn"
Sep 16 07:54:39.070472 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.070301    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn" event={"ID":"94a7acdd-045d-408d-81e9-5003dd696975","Type":"ContainerDied","Data":"eb2027dbdd05919d60eceeddc7e869fbb930363ff5b2ef1cc024634728ff5f17"}
Sep 16 07:54:39.070472 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.070350    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb2027dbdd05919d60eceeddc7e869fbb930363ff5b2ef1cc024634728ff5f17"
Sep 16 07:54:39.684198 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.684169    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:54:39.851528 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.851456    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-workload-socket\") pod \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") "
Sep 16 07:54:39.851528 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.851496    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-token\") pod \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") "
Sep 16 07:54:39.851737 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.851544    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-data\") pod \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") "
Sep 16 07:54:39.851737 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.851565    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-podinfo\") pod \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") "
Sep 16 07:54:39.851737 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.851589    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdzt4\" (UniqueName: \"kubernetes.io/projected/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-kube-api-access-hdzt4\") pod \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") "
Sep 16 07:54:39.851737 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.851640    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istiod-ca-cert\") pod \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") "
Sep 16 07:54:39.851737 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.851671    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-credential-socket\") pod \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") "
Sep 16 07:54:39.851737 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.851720    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-envoy\") pod \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") "
Sep 16 07:54:39.852094 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.851720    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" (UID: "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:54:39.852094 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.851763    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-workload-certs\") pod \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\" (UID: \"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236\") "
Sep 16 07:54:39.852094 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.851798    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" (UID: "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:54:39.852094 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.851978    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" (UID: "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:54:39.852094 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.852080    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" (UID: "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:54:39.852332 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.852096    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" (UID: "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:54:39.852332 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.852208    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:54:39.852332 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.852225    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:54:39.852332 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.852238    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:54:39.852332 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.852251    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:54:39.852332 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.852263    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:54:39.853745 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.853715    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" (UID: "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 07:54:39.853873 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.853790    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" (UID: "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:54:39.854165 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.854138    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" (UID: "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:54:39.854165 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.854154    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-kube-api-access-hdzt4" (OuterVolumeSpecName: "kube-api-access-hdzt4") pod "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" (UID: "7b309cf1-ec9d-4d4d-808b-ef1f4d54a236"). InnerVolumeSpecName "kube-api-access-hdzt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:54:39.952982 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.952956    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdzt4\" (UniqueName: \"kubernetes.io/projected/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-kube-api-access-hdzt4\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:54:39.952982 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.952979    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:54:39.953124 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.952989    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:54:39.953124 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:39.952997    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:54:40.077260 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:40.077228    2581 generic.go:356] "Generic (PLEG): container finished" podID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerID="1f8b616d3dea9802949515cdd5a1baec743317cf2dbfac3368ce54c9f25a2fce" exitCode=0
Sep 16 07:54:40.077426 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:40.077271    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"
Sep 16 07:54:40.077426 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:40.077281    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp" event={"ID":"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236","Type":"ContainerDied","Data":"1f8b616d3dea9802949515cdd5a1baec743317cf2dbfac3368ce54c9f25a2fce"}
Sep 16 07:54:40.077426 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:40.077328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp" event={"ID":"7b309cf1-ec9d-4d4d-808b-ef1f4d54a236","Type":"ContainerDied","Data":"379d808fb4142b2e9adea9a26d5ac2d0830219405c9f9ca49e52df1034bb2f3f"}
Sep 16 07:54:40.077426 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:40.077350    2581 scope.go:117] "RemoveContainer" containerID="1f8b616d3dea9802949515cdd5a1baec743317cf2dbfac3368ce54c9f25a2fce"
Sep 16 07:54:40.087315 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:40.087299    2581 scope.go:117] "RemoveContainer" containerID="68262c4d2c267115c5fb3dd0a0ee9d5e34e638124e1cec17f0a17c3da697dd7c"
Sep 16 07:54:40.095362 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:40.095343    2581 scope.go:117] "RemoveContainer" containerID="1f8b616d3dea9802949515cdd5a1baec743317cf2dbfac3368ce54c9f25a2fce"
Sep 16 07:54:40.095639 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:54:40.095622    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8b616d3dea9802949515cdd5a1baec743317cf2dbfac3368ce54c9f25a2fce\": container with ID starting with 1f8b616d3dea9802949515cdd5a1baec743317cf2dbfac3368ce54c9f25a2fce not found: ID does not exist" containerID="1f8b616d3dea9802949515cdd5a1baec743317cf2dbfac3368ce54c9f25a2fce"
Sep 16 07:54:40.095686 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:40.095649    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8b616d3dea9802949515cdd5a1baec743317cf2dbfac3368ce54c9f25a2fce"} err="failed to get container status \"1f8b616d3dea9802949515cdd5a1baec743317cf2dbfac3368ce54c9f25a2fce\": rpc error: code = NotFound desc = could not find container \"1f8b616d3dea9802949515cdd5a1baec743317cf2dbfac3368ce54c9f25a2fce\": container with ID starting with 1f8b616d3dea9802949515cdd5a1baec743317cf2dbfac3368ce54c9f25a2fce not found: ID does not exist"
Sep 16 07:54:40.095686 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:40.095664    2581 scope.go:117] "RemoveContainer" containerID="68262c4d2c267115c5fb3dd0a0ee9d5e34e638124e1cec17f0a17c3da697dd7c"
Sep 16 07:54:40.095899 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:54:40.095881    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68262c4d2c267115c5fb3dd0a0ee9d5e34e638124e1cec17f0a17c3da697dd7c\": container with ID starting with 68262c4d2c267115c5fb3dd0a0ee9d5e34e638124e1cec17f0a17c3da697dd7c not found: ID does not exist" containerID="68262c4d2c267115c5fb3dd0a0ee9d5e34e638124e1cec17f0a17c3da697dd7c"
Sep 16 07:54:40.095952 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:40.095906    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68262c4d2c267115c5fb3dd0a0ee9d5e34e638124e1cec17f0a17c3da697dd7c"} err="failed to get container status \"68262c4d2c267115c5fb3dd0a0ee9d5e34e638124e1cec17f0a17c3da697dd7c\": rpc error: code = NotFound desc = could not find container \"68262c4d2c267115c5fb3dd0a0ee9d5e34e638124e1cec17f0a17c3da697dd7c\": container with ID starting with 68262c4d2c267115c5fb3dd0a0ee9d5e34e638124e1cec17f0a17c3da697dd7c not found: ID does not exist"
Sep 16 07:54:40.099663 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:40.099643    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"]
Sep 16 07:54:40.102722 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:40.102677    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5dd30f-openshift-default-66f4b9fc98-thzpp"]
Sep 16 07:54:41.684442 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:41.684402    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" path="/var/lib/kubelet/pods/7b309cf1-ec9d-4d4d-808b-ef1f4d54a236/volumes"
Sep 16 07:54:45.684423 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:45.684394    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn"]
Sep 16 07:54:45.684423 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:45.684430    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dd359c-wm5tn"]
Sep 16 07:54:47.689722 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:47.689638    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94a7acdd-045d-408d-81e9-5003dd696975" path="/var/lib/kubelet/pods/94a7acdd-045d-408d-81e9-5003dd696975/volumes"
Sep 16 07:54:51.021038 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:51.021004    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"]
Sep 16 07:54:59.676932 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:54:59.676849    2581 scope.go:117] "RemoveContainer" containerID="c9710708bcb1bc311b5d6c1b3a8507ee150f0a4350419c0dd7e5c07a9e114777"
Sep 16 07:55:06.040399 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.040349    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj" podUID="2013af00-9176-4429-8d9c-9e977456d921" containerName="main" containerID="cri-o://cf823f8b0cc3d5be4d6791620d54d8ed402acc713a5e86b53319404f755b2641" gracePeriod=45
Sep 16 07:55:06.235169 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.235133    2581 generic.go:356] "Generic (PLEG): container finished" podID="2013af00-9176-4429-8d9c-9e977456d921" containerID="cf823f8b0cc3d5be4d6791620d54d8ed402acc713a5e86b53319404f755b2641" exitCode=0
Sep 16 07:55:06.235356 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.235199    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj" event={"ID":"2013af00-9176-4429-8d9c-9e977456d921","Type":"ContainerDied","Data":"cf823f8b0cc3d5be4d6791620d54d8ed402acc713a5e86b53319404f755b2641"}
Sep 16 07:55:06.296036 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.295974    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:55:06.413406 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.413373    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-dshm\") pod \"2013af00-9176-4429-8d9c-9e977456d921\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") "
Sep 16 07:55:06.413589 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.413468    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8gkf\" (UniqueName: \"kubernetes.io/projected/2013af00-9176-4429-8d9c-9e977456d921-kube-api-access-n8gkf\") pod \"2013af00-9176-4429-8d9c-9e977456d921\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") "
Sep 16 07:55:06.413589 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.413495    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2013af00-9176-4429-8d9c-9e977456d921-tls-certs\") pod \"2013af00-9176-4429-8d9c-9e977456d921\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") "
Sep 16 07:55:06.413589 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.413526    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-tmp-dir\") pod \"2013af00-9176-4429-8d9c-9e977456d921\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") "
Sep 16 07:55:06.413589 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.413551    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-model-cache\") pod \"2013af00-9176-4429-8d9c-9e977456d921\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") "
Sep 16 07:55:06.413829 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.413639    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-home\") pod \"2013af00-9176-4429-8d9c-9e977456d921\" (UID: \"2013af00-9176-4429-8d9c-9e977456d921\") "
Sep 16 07:55:06.414015 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.413935    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-home" (OuterVolumeSpecName: "home") pod "2013af00-9176-4429-8d9c-9e977456d921" (UID: "2013af00-9176-4429-8d9c-9e977456d921"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:55:06.414015 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.413951    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-model-cache" (OuterVolumeSpecName: "model-cache") pod "2013af00-9176-4429-8d9c-9e977456d921" (UID: "2013af00-9176-4429-8d9c-9e977456d921"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:55:06.414194 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.414015    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2013af00-9176-4429-8d9c-9e977456d921" (UID: "2013af00-9176-4429-8d9c-9e977456d921"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:55:06.414194 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.414115    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:55:06.414194 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.414133    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:55:06.414194 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.414145    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:55:06.416212 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.416186    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-dshm" (OuterVolumeSpecName: "dshm") pod "2013af00-9176-4429-8d9c-9e977456d921" (UID: "2013af00-9176-4429-8d9c-9e977456d921"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:55:06.416296 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.416223    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2013af00-9176-4429-8d9c-9e977456d921-kube-api-access-n8gkf" (OuterVolumeSpecName: "kube-api-access-n8gkf") pod "2013af00-9176-4429-8d9c-9e977456d921" (UID: "2013af00-9176-4429-8d9c-9e977456d921"). InnerVolumeSpecName "kube-api-access-n8gkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:55:06.416296 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.416240    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2013af00-9176-4429-8d9c-9e977456d921-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2013af00-9176-4429-8d9c-9e977456d921" (UID: "2013af00-9176-4429-8d9c-9e977456d921"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 07:55:06.514998 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.514959    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2013af00-9176-4429-8d9c-9e977456d921-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:55:06.514998 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.514990    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2013af00-9176-4429-8d9c-9e977456d921-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:55:06.514998 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:06.515001    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8gkf\" (UniqueName: \"kubernetes.io/projected/2013af00-9176-4429-8d9c-9e977456d921-kube-api-access-n8gkf\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:55:07.242464 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:07.242435    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"
Sep 16 07:55:07.242929 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:07.242472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj" event={"ID":"2013af00-9176-4429-8d9c-9e977456d921","Type":"ContainerDied","Data":"5d3f6f7aa9566289c9498b5965ee113bf36eb4747bf6169c40db4fd0b8cc66bb"}
Sep 16 07:55:07.242929 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:07.242524    2581 scope.go:117] "RemoveContainer" containerID="cf823f8b0cc3d5be4d6791620d54d8ed402acc713a5e86b53319404f755b2641"
Sep 16 07:55:07.265240 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:07.265214    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"]
Sep 16 07:55:07.268912 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:07.268889    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-dd359c/test-model-dd359c-kserve-6f6bb7995c-4m7lj"]
Sep 16 07:55:07.683932 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:07.683901    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2013af00-9176-4429-8d9c-9e977456d921" path="/var/lib/kubelet/pods/2013af00-9176-4429-8d9c-9e977456d921/volumes"
Sep 16 07:55:32.175434 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.175400    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"]
Sep 16 07:55:32.175934 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.175847    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2013af00-9176-4429-8d9c-9e977456d921" containerName="main"
Sep 16 07:55:32.175934 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.175863    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2013af00-9176-4429-8d9c-9e977456d921" containerName="main"
Sep 16 07:55:32.175934 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.175877    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerName="istio-proxy"
Sep 16 07:55:32.175934 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.175882    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerName="istio-proxy"
Sep 16 07:55:32.175934 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.175892    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a7acdd-045d-408d-81e9-5003dd696975" containerName="revoke-keys"
Sep 16 07:55:32.175934 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.175897    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a7acdd-045d-408d-81e9-5003dd696975" containerName="revoke-keys"
Sep 16 07:55:32.175934 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.175911    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerName="istio-proxy"
Sep 16 07:55:32.175934 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.175915    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerName="istio-proxy"
Sep 16 07:55:32.176384 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.175972    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerName="istio-proxy"
Sep 16 07:55:32.176384 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.175980    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2013af00-9176-4429-8d9c-9e977456d921" containerName="main"
Sep 16 07:55:32.176384 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.175988    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b309cf1-ec9d-4d4d-808b-ef1f4d54a236" containerName="istio-proxy"
Sep 16 07:55:32.176384 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.175996    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="94a7acdd-045d-408d-81e9-5003dd696975" containerName="revoke-keys"
Sep 16 07:55:32.178848 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.178802    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.181164 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.181145    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"e2e-conflict-b-8a3fd4d9-kserve-self-signed-certs\""
Sep 16 07:55:32.189422 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.189398    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"]
Sep 16 07:55:32.272413 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.272375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqmdq\" (UniqueName: \"kubernetes.io/projected/bd98f9a4-5bff-4759-a394-8935e1e304f8-kube-api-access-vqmdq\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.272680 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.272465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-home\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.272680 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.272533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-dshm\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.272680 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.272592    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-tmp-dir\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.272680 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.272622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-model-cache\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.273042 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.272687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd98f9a4-5bff-4759-a394-8935e1e304f8-tls-certs\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.373920 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.373887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd98f9a4-5bff-4759-a394-8935e1e304f8-tls-certs\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.374100 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.373932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqmdq\" (UniqueName: \"kubernetes.io/projected/bd98f9a4-5bff-4759-a394-8935e1e304f8-kube-api-access-vqmdq\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.374100 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.373975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-home\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.374100 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.374012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-dshm\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.374100 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.374044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-tmp-dir\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.374100 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.374069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-model-cache\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.374493 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.374460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-home\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.374493 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.374482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-tmp-dir\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.374653 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.374534    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-model-cache\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.376268 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.376235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-dshm\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.376630 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.376610    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd98f9a4-5bff-4759-a394-8935e1e304f8-tls-certs\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.382397 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.382371    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqmdq\" (UniqueName: \"kubernetes.io/projected/bd98f9a4-5bff-4759-a394-8935e1e304f8-kube-api-access-vqmdq\") pod \"e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") " pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.490002 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.489921    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:32.785537 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:32.785508    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"]
Sep 16 07:55:32.786833 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:55:32.786790    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd98f9a4_5bff_4759_a394_8935e1e304f8.slice/crio-4ecf05b5bd08179a92fc946b16a9161ebfc4cb6c390292fa99a296e86d840a5f WatchSource:0}: Error finding container 4ecf05b5bd08179a92fc946b16a9161ebfc4cb6c390292fa99a296e86d840a5f: Status 404 returned error can't find the container with id 4ecf05b5bd08179a92fc946b16a9161ebfc4cb6c390292fa99a296e86d840a5f
Sep 16 07:55:33.391050 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:33.391010    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s" event={"ID":"bd98f9a4-5bff-4759-a394-8935e1e304f8","Type":"ContainerStarted","Data":"0e6f741f6f2e2390978ccd0819eec84247556d09ca1498b32b39e40c73f06d28"}
Sep 16 07:55:33.391394 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:33.391056    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s" event={"ID":"bd98f9a4-5bff-4759-a394-8935e1e304f8","Type":"ContainerStarted","Data":"4ecf05b5bd08179a92fc946b16a9161ebfc4cb6c390292fa99a296e86d840a5f"}
Sep 16 07:55:33.407917 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:33.407869    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s" podStartSLOduration=1.407856163 podStartE2EDuration="1.407856163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:55:33.406539253 +0000 UTC m=+1294.333151406" watchObservedRunningTime="2026-09-16 07:55:33.407856163 +0000 UTC m=+1294.334468315"
Sep 16 07:55:34.825546 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.825511    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"]
Sep 16 07:55:34.829116 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.829094    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.831876 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.831855    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-3616cc-openshift-default-dockercfg-gv5tt\""
Sep 16 07:55:34.839645 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.839626    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"]
Sep 16 07:55:34.898193 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.898163    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-token\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.898310 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.898204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp6r6\" (UniqueName: \"kubernetes.io/projected/514f0b71-05b8-4c88-8baf-2f2667cd22d4-kube-api-access-bp6r6\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.898310 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.898238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-podinfo\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.898310 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.898267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-envoy\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.898419 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.898371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-workload-socket\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.898419 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.898413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-workload-certs\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.898488 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.898438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-data\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.898532 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.898508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-credential-socket\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.898575 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.898558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istiod-ca-cert\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.999480 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.999441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-credential-socket\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.999682 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.999501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istiod-ca-cert\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.999682 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.999544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-token\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.999682 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.999567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp6r6\" (UniqueName: \"kubernetes.io/projected/514f0b71-05b8-4c88-8baf-2f2667cd22d4-kube-api-access-bp6r6\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.999682 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.999601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-podinfo\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.999682 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.999629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-envoy\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.999971 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.999731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-workload-socket\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.999971 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.999781    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-workload-certs\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.999971 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.999808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-data\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:34.999971 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:34.999925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-credential-socket\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:35.000200 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:35.000163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-data\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:35.000311 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:35.000289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istiod-ca-cert\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:35.000396 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:35.000375    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-workload-certs\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:35.000497 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:35.000418    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-workload-socket\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:35.002065 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:35.002042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-envoy\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:35.002290 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:35.002269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-podinfo\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:35.007688 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:35.007657    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp6r6\" (UniqueName: \"kubernetes.io/projected/514f0b71-05b8-4c88-8baf-2f2667cd22d4-kube-api-access-bp6r6\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:35.007800 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:35.007708    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-token\") pod \"e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") " pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:35.143173 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:35.143098    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:35.423998 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:35.423973    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"]
Sep 16 07:55:35.425810 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:55:35.425779    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514f0b71_05b8_4c88_8baf_2f2667cd22d4.slice/crio-88d614a93ea6043fe0e1fcd4aca7b38d2a2dfd33dbae3f412bb4179333fbb52b WatchSource:0}: Error finding container 88d614a93ea6043fe0e1fcd4aca7b38d2a2dfd33dbae3f412bb4179333fbb52b: Status 404 returned error can't find the container with id 88d614a93ea6043fe0e1fcd4aca7b38d2a2dfd33dbae3f412bb4179333fbb52b
Sep 16 07:55:35.428091 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:35.428060    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:55:35.428184 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:35.428120    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:55:35.428184 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:35.428148    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:55:36.420244 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:36.420211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g" event={"ID":"514f0b71-05b8-4c88-8baf-2f2667cd22d4","Type":"ContainerStarted","Data":"8c1ded9a21e07440f49404d8c434fdb02a34d390c4dc466f4ac963f0945f24f7"}
Sep 16 07:55:36.420712 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:36.420298    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g" event={"ID":"514f0b71-05b8-4c88-8baf-2f2667cd22d4","Type":"ContainerStarted","Data":"88d614a93ea6043fe0e1fcd4aca7b38d2a2dfd33dbae3f412bb4179333fbb52b"}
Sep 16 07:55:36.442011 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:36.441963    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g" podStartSLOduration=2.441947824 podStartE2EDuration="2.441947824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:55:36.440391541 +0000 UTC m=+1297.367003694" watchObservedRunningTime="2026-09-16 07:55:36.441947824 +0000 UTC m=+1297.368559976"
Sep 16 07:55:37.144154 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:37.144119    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:37.149364 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:37.149338    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:37.425651 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:37.425623    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:37.426531 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:37.426514    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:55:42.490722 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:42.490671    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:42.491272 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:42.490737    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:42.505063 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:42.505037    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:43.469835 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:43.469789    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:55:59.709193 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:59.709163    2581 scope.go:117] "RemoveContainer" containerID="b9b527fc10d2ba274438227cef48c23b5f39f21f1feeb88df89ff8bdcd76a963"
Sep 16 07:55:59.724872 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:55:59.724846    2581 scope.go:117] "RemoveContainer" containerID="cd4d6418f782f6442b43379b986c9cfba25a4d21879669698521d174a667935c"
Sep 16 07:56:06.590747 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:06.590715    2581 generic.go:356] "Generic (PLEG): container finished" podID="514f0b71-05b8-4c88-8baf-2f2667cd22d4" containerID="8c1ded9a21e07440f49404d8c434fdb02a34d390c4dc466f4ac963f0945f24f7" exitCode=0
Sep 16 07:56:06.591208 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:06.590791    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g" event={"ID":"514f0b71-05b8-4c88-8baf-2f2667cd22d4","Type":"ContainerDied","Data":"8c1ded9a21e07440f49404d8c434fdb02a34d390c4dc466f4ac963f0945f24f7"}
Sep 16 07:56:06.591254 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:06.591240    2581 scope.go:117] "RemoveContainer" containerID="8c1ded9a21e07440f49404d8c434fdb02a34d390c4dc466f4ac963f0945f24f7"
Sep 16 07:56:06.592524 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:06.592497    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:56:06.592586 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:06.592562    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:56:06.592621 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:06.592595    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:56:07.425953 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:07.425916    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:56:07.599691 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:07.599656    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g" event={"ID":"514f0b71-05b8-4c88-8baf-2f2667cd22d4","Type":"ContainerStarted","Data":"d5b7d1c43c0ea9977251230393ca7691c155f3bbeec24b550b8afc381adc4800"}
Sep 16 07:56:08.143379 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:08.143344    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:56:08.144403 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:08.144377    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 16 07:56:08.144526 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:08.144434    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g" podUID="514f0b71-05b8-4c88-8baf-2f2667cd22d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 16 07:56:09.143873 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:09.143835    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 16 07:56:09.144299 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:09.143909    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g" podUID="514f0b71-05b8-4c88-8baf-2f2667cd22d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 16 07:56:10.143441 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:10.143406    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 16 07:56:10.143616 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:10.143496    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g" podUID="514f0b71-05b8-4c88-8baf-2f2667cd22d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 16 07:56:11.144398 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:11.144361    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 16 07:56:11.144788 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:11.144418    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g" podUID="514f0b71-05b8-4c88-8baf-2f2667cd22d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 16 07:56:12.143468 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:12.143422    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 16 07:56:12.143641 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:12.143490    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g" podUID="514f0b71-05b8-4c88-8baf-2f2667cd22d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 16 07:56:13.147444 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:13.147412    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:56:13.147852 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:13.147684    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:56:13.148302 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:13.148285    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 07:56:14.198427 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.198384    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"]
Sep 16 07:56:14.199536 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.199469    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt" podUID="9671b553-9409-4397-9cfb-7cf62cf76515" containerName="payload-processing" containerID="cri-o://0004b44f4ee1c454aa15741d2aeb17dc987d62f7d1f8302dd75ccc918f7db6fb" gracePeriod=30
Sep 16 07:56:14.504924 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.504874    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"
Sep 16 07:56:14.617427 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.617392    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9671b553-9409-4397-9cfb-7cf62cf76515-plugins-config-volume\") pod \"9671b553-9409-4397-9cfb-7cf62cf76515\" (UID: \"9671b553-9409-4397-9cfb-7cf62cf76515\") "
Sep 16 07:56:14.617595 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.617515    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrc9p\" (UniqueName: \"kubernetes.io/projected/9671b553-9409-4397-9cfb-7cf62cf76515-kube-api-access-wrc9p\") pod \"9671b553-9409-4397-9cfb-7cf62cf76515\" (UID: \"9671b553-9409-4397-9cfb-7cf62cf76515\") "
Sep 16 07:56:14.617809 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.617786    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9671b553-9409-4397-9cfb-7cf62cf76515-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9671b553-9409-4397-9cfb-7cf62cf76515" (UID: "9671b553-9409-4397-9cfb-7cf62cf76515"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:56:14.617965 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.617951    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9671b553-9409-4397-9cfb-7cf62cf76515-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:56:14.619624 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.619597    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9671b553-9409-4397-9cfb-7cf62cf76515-kube-api-access-wrc9p" (OuterVolumeSpecName: "kube-api-access-wrc9p") pod "9671b553-9409-4397-9cfb-7cf62cf76515" (UID: "9671b553-9409-4397-9cfb-7cf62cf76515"). InnerVolumeSpecName "kube-api-access-wrc9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:56:14.655289 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.655258    2581 generic.go:356] "Generic (PLEG): container finished" podID="9671b553-9409-4397-9cfb-7cf62cf76515" containerID="0004b44f4ee1c454aa15741d2aeb17dc987d62f7d1f8302dd75ccc918f7db6fb" exitCode=0
Sep 16 07:56:14.655429 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.655308    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"
Sep 16 07:56:14.655429 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.655347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt" event={"ID":"9671b553-9409-4397-9cfb-7cf62cf76515","Type":"ContainerDied","Data":"0004b44f4ee1c454aa15741d2aeb17dc987d62f7d1f8302dd75ccc918f7db6fb"}
Sep 16 07:56:14.655429 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.655393    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt" event={"ID":"9671b553-9409-4397-9cfb-7cf62cf76515","Type":"ContainerDied","Data":"d144ecb28dd5ba12358d943e71422822e5849bc1ba79119b82904e987d8691f2"}
Sep 16 07:56:14.655429 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.655416    2581 scope.go:117] "RemoveContainer" containerID="0004b44f4ee1c454aa15741d2aeb17dc987d62f7d1f8302dd75ccc918f7db6fb"
Sep 16 07:56:14.666759 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.666743    2581 scope.go:117] "RemoveContainer" containerID="0004b44f4ee1c454aa15741d2aeb17dc987d62f7d1f8302dd75ccc918f7db6fb"
Sep 16 07:56:14.667129 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:56:14.667110    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0004b44f4ee1c454aa15741d2aeb17dc987d62f7d1f8302dd75ccc918f7db6fb\": container with ID starting with 0004b44f4ee1c454aa15741d2aeb17dc987d62f7d1f8302dd75ccc918f7db6fb not found: ID does not exist" containerID="0004b44f4ee1c454aa15741d2aeb17dc987d62f7d1f8302dd75ccc918f7db6fb"
Sep 16 07:56:14.667177 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.667139    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0004b44f4ee1c454aa15741d2aeb17dc987d62f7d1f8302dd75ccc918f7db6fb"} err="failed to get container status \"0004b44f4ee1c454aa15741d2aeb17dc987d62f7d1f8302dd75ccc918f7db6fb\": rpc error: code = NotFound desc = could not find container \"0004b44f4ee1c454aa15741d2aeb17dc987d62f7d1f8302dd75ccc918f7db6fb\": container with ID starting with 0004b44f4ee1c454aa15741d2aeb17dc987d62f7d1f8302dd75ccc918f7db6fb not found: ID does not exist"
Sep 16 07:56:14.679611 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.679590    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"]
Sep 16 07:56:14.681748 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.681729    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc2a294d-fdc665f65-9jpbt"]
Sep 16 07:56:14.719496 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:14.719439    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrc9p\" (UniqueName: \"kubernetes.io/projected/9671b553-9409-4397-9cfb-7cf62cf76515-kube-api-access-wrc9p\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:56:15.683439 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:15.683405    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9671b553-9409-4397-9cfb-7cf62cf76515" path="/var/lib/kubelet/pods/9671b553-9409-4397-9cfb-7cf62cf76515/volumes"
Sep 16 07:56:17.281127 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.281096    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"]
Sep 16 07:56:17.281531 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.281518    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9671b553-9409-4397-9cfb-7cf62cf76515" containerName="payload-processing"
Sep 16 07:56:17.281531 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.281533    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9671b553-9409-4397-9cfb-7cf62cf76515" containerName="payload-processing"
Sep 16 07:56:17.281607 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.281600    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9671b553-9409-4397-9cfb-7cf62cf76515" containerName="payload-processing"
Sep 16 07:56:17.284445 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.284426    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"
Sep 16 07:56:17.287058 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.287021    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e9fc8c\""
Sep 16 07:56:17.287179 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.287162    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e9fc8c-dockercfg-rxzrb\""
Sep 16 07:56:17.294274 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.294251    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"]
Sep 16 07:56:17.447450 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.447417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm9dn\" (UniqueName: \"kubernetes.io/projected/f499f35e-d8a6-4f67-b3f9-f9377e6d96a3-kube-api-access-rm9dn\") pod \"payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p\" (UID: \"f499f35e-d8a6-4f67-b3f9-f9377e6d96a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"
Sep 16 07:56:17.447613 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.447566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f499f35e-d8a6-4f67-b3f9-f9377e6d96a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p\" (UID: \"f499f35e-d8a6-4f67-b3f9-f9377e6d96a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"
Sep 16 07:56:17.548806 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.548727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm9dn\" (UniqueName: \"kubernetes.io/projected/f499f35e-d8a6-4f67-b3f9-f9377e6d96a3-kube-api-access-rm9dn\") pod \"payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p\" (UID: \"f499f35e-d8a6-4f67-b3f9-f9377e6d96a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"
Sep 16 07:56:17.548973 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.548863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f499f35e-d8a6-4f67-b3f9-f9377e6d96a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p\" (UID: \"f499f35e-d8a6-4f67-b3f9-f9377e6d96a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"
Sep 16 07:56:17.549557 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.549533    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f499f35e-d8a6-4f67-b3f9-f9377e6d96a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p\" (UID: \"f499f35e-d8a6-4f67-b3f9-f9377e6d96a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"
Sep 16 07:56:17.556574 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.556547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm9dn\" (UniqueName: \"kubernetes.io/projected/f499f35e-d8a6-4f67-b3f9-f9377e6d96a3-kube-api-access-rm9dn\") pod \"payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p\" (UID: \"f499f35e-d8a6-4f67-b3f9-f9377e6d96a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"
Sep 16 07:56:17.595963 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.595938    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"
Sep 16 07:56:17.869306 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:17.869284    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"]
Sep 16 07:56:17.871020 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:56:17.870995    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf499f35e_d8a6_4f67_b3f9_f9377e6d96a3.slice/crio-164e1e649f354b812256d510b56ed971037fcbe497deda7fcc0c816b7572c0dc WatchSource:0}: Error finding container 164e1e649f354b812256d510b56ed971037fcbe497deda7fcc0c816b7572c0dc: Status 404 returned error can't find the container with id 164e1e649f354b812256d510b56ed971037fcbe497deda7fcc0c816b7572c0dc
Sep 16 07:56:18.680311 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:18.680271    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p" event={"ID":"f499f35e-d8a6-4f67-b3f9-f9377e6d96a3","Type":"ContainerStarted","Data":"26fafa86cff821900ebf246896d90f8ee8fca1112b199da8b096b5c0ee44d233"}
Sep 16 07:56:18.680692 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:18.680319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p" event={"ID":"f499f35e-d8a6-4f67-b3f9-f9377e6d96a3","Type":"ContainerStarted","Data":"164e1e649f354b812256d510b56ed971037fcbe497deda7fcc0c816b7572c0dc"}
Sep 16 07:56:18.680692 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:18.680359    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"
Sep 16 07:56:18.698934 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:18.698890    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p" podStartSLOduration=1.698875203 podStartE2EDuration="1.698875203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:56:18.696518849 +0000 UTC m=+1339.623131004" watchObservedRunningTime="2026-09-16 07:56:18.698875203 +0000 UTC m=+1339.625487356"
Sep 16 07:56:29.688945 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:29.688738    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"
Sep 16 07:56:37.088483 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:37.088445    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"]
Sep 16 07:56:38.961353 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:38.961318    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"]
Sep 16 07:56:52.111462 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.111399    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s" podUID="bd98f9a4-5bff-4759-a394-8935e1e304f8" containerName="main" containerID="cri-o://0e6f741f6f2e2390978ccd0819eec84247556d09ca1498b32b39e40c73f06d28" gracePeriod=45
Sep 16 07:56:52.357382 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.357359    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:56:52.494915 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.494887    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-dshm\") pod \"bd98f9a4-5bff-4759-a394-8935e1e304f8\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") "
Sep 16 07:56:52.495089 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.494920    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqmdq\" (UniqueName: \"kubernetes.io/projected/bd98f9a4-5bff-4759-a394-8935e1e304f8-kube-api-access-vqmdq\") pod \"bd98f9a4-5bff-4759-a394-8935e1e304f8\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") "
Sep 16 07:56:52.495089 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.495003    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd98f9a4-5bff-4759-a394-8935e1e304f8-tls-certs\") pod \"bd98f9a4-5bff-4759-a394-8935e1e304f8\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") "
Sep 16 07:56:52.495089 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.495019    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-tmp-dir\") pod \"bd98f9a4-5bff-4759-a394-8935e1e304f8\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") "
Sep 16 07:56:52.495285 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.495152    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-home\") pod \"bd98f9a4-5bff-4759-a394-8935e1e304f8\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") "
Sep 16 07:56:52.495285 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.495238    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-model-cache\") pod \"bd98f9a4-5bff-4759-a394-8935e1e304f8\" (UID: \"bd98f9a4-5bff-4759-a394-8935e1e304f8\") "
Sep 16 07:56:52.495285 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.495244    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd98f9a4-5bff-4759-a394-8935e1e304f8" (UID: "bd98f9a4-5bff-4759-a394-8935e1e304f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:56:52.495451 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.495399    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-home" (OuterVolumeSpecName: "home") pod "bd98f9a4-5bff-4759-a394-8935e1e304f8" (UID: "bd98f9a4-5bff-4759-a394-8935e1e304f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:56:52.495502 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.495464    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd98f9a4-5bff-4759-a394-8935e1e304f8" (UID: "bd98f9a4-5bff-4759-a394-8935e1e304f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:56:52.495659 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.495643    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:56:52.495698 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.495661    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:56:52.495698 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.495670    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:56:52.497164 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.497134    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd98f9a4-5bff-4759-a394-8935e1e304f8-kube-api-access-vqmdq" (OuterVolumeSpecName: "kube-api-access-vqmdq") pod "bd98f9a4-5bff-4759-a394-8935e1e304f8" (UID: "bd98f9a4-5bff-4759-a394-8935e1e304f8"). InnerVolumeSpecName "kube-api-access-vqmdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:56:52.497319 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.497302    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-dshm" (OuterVolumeSpecName: "dshm") pod "bd98f9a4-5bff-4759-a394-8935e1e304f8" (UID: "bd98f9a4-5bff-4759-a394-8935e1e304f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:56:52.497482 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.497461    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd98f9a4-5bff-4759-a394-8935e1e304f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd98f9a4-5bff-4759-a394-8935e1e304f8" (UID: "bd98f9a4-5bff-4759-a394-8935e1e304f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 07:56:52.596901 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.596875    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqmdq\" (UniqueName: \"kubernetes.io/projected/bd98f9a4-5bff-4759-a394-8935e1e304f8-kube-api-access-vqmdq\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:56:52.596901 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.596899    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd98f9a4-5bff-4759-a394-8935e1e304f8-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:56:52.597041 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.596910    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd98f9a4-5bff-4759-a394-8935e1e304f8-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:56:52.858572 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.858502    2581 generic.go:356] "Generic (PLEG): container finished" podID="bd98f9a4-5bff-4759-a394-8935e1e304f8" containerID="0e6f741f6f2e2390978ccd0819eec84247556d09ca1498b32b39e40c73f06d28" exitCode=0
Sep 16 07:56:52.858572 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.858535    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"
Sep 16 07:56:52.858760 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.858534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s" event={"ID":"bd98f9a4-5bff-4759-a394-8935e1e304f8","Type":"ContainerDied","Data":"0e6f741f6f2e2390978ccd0819eec84247556d09ca1498b32b39e40c73f06d28"}
Sep 16 07:56:52.858760 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.858676    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s" event={"ID":"bd98f9a4-5bff-4759-a394-8935e1e304f8","Type":"ContainerDied","Data":"4ecf05b5bd08179a92fc946b16a9161ebfc4cb6c390292fa99a296e86d840a5f"}
Sep 16 07:56:52.858760 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.858703    2581 scope.go:117] "RemoveContainer" containerID="0e6f741f6f2e2390978ccd0819eec84247556d09ca1498b32b39e40c73f06d28"
Sep 16 07:56:52.868188 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.868173    2581 scope.go:117] "RemoveContainer" containerID="0e6f741f6f2e2390978ccd0819eec84247556d09ca1498b32b39e40c73f06d28"
Sep 16 07:56:52.868442 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:56:52.868425    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e6f741f6f2e2390978ccd0819eec84247556d09ca1498b32b39e40c73f06d28\": container with ID starting with 0e6f741f6f2e2390978ccd0819eec84247556d09ca1498b32b39e40c73f06d28 not found: ID does not exist" containerID="0e6f741f6f2e2390978ccd0819eec84247556d09ca1498b32b39e40c73f06d28"
Sep 16 07:56:52.868501 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.868453    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e6f741f6f2e2390978ccd0819eec84247556d09ca1498b32b39e40c73f06d28"} err="failed to get container status \"0e6f741f6f2e2390978ccd0819eec84247556d09ca1498b32b39e40c73f06d28\": rpc error: code = NotFound desc = could not find container \"0e6f741f6f2e2390978ccd0819eec84247556d09ca1498b32b39e40c73f06d28\": container with ID starting with 0e6f741f6f2e2390978ccd0819eec84247556d09ca1498b32b39e40c73f06d28 not found: ID does not exist"
Sep 16 07:56:52.879795 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.879773    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"]
Sep 16 07:56:52.889291 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:52.885053    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-b-8a3fd4d9-kserve-79dbcbc47b-q4m9s"]
Sep 16 07:56:53.469792 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.469758    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"]
Sep 16 07:56:53.470241 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.470227    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd98f9a4-5bff-4759-a394-8935e1e304f8" containerName="main"
Sep 16 07:56:53.470288 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.470243    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd98f9a4-5bff-4759-a394-8935e1e304f8" containerName="main"
Sep 16 07:56:53.470334 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.470308    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd98f9a4-5bff-4759-a394-8935e1e304f8" containerName="main"
Sep 16 07:56:53.475086 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.475069    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.478622 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.478597    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3616cc\"/\"kube-root-ca.crt\""
Sep 16 07:56:53.478845 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.478605    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3616cc\"/\"auth-test-model-3616cc-kserve-self-signed-certs\""
Sep 16 07:56:53.479229 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.479038    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3616cc\"/\"openshift-service-ca.crt\""
Sep 16 07:56:53.479500 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.479477    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3616cc\"/\"default-dockercfg-lm9f7\""
Sep 16 07:56:53.481545 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.481521    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"]
Sep 16 07:56:53.608512 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.608479    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qcb9\" (UniqueName: \"kubernetes.io/projected/328dfdf4-6723-4d54-8902-2d217f6be6a4-kube-api-access-8qcb9\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.608684 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.608541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-dshm\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.608684 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.608616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-tmp-dir\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.608684 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.608645    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328dfdf4-6723-4d54-8902-2d217f6be6a4-tls-certs\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.608876 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.608686    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-home\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.608876 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.608798    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-model-cache\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.697848 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.697079    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd98f9a4-5bff-4759-a394-8935e1e304f8" path="/var/lib/kubelet/pods/bd98f9a4-5bff-4759-a394-8935e1e304f8/volumes"
Sep 16 07:56:53.709644 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.709527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-model-cache\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.709644 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.709622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qcb9\" (UniqueName: \"kubernetes.io/projected/328dfdf4-6723-4d54-8902-2d217f6be6a4-kube-api-access-8qcb9\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.709964 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.709689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-dshm\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.709964 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.709753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-tmp-dir\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.709964 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.709782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328dfdf4-6723-4d54-8902-2d217f6be6a4-tls-certs\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.709964 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.709843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-home\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.711219 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.710288    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-home\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.711219 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.710511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-model-cache\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.712371 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.712333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-tmp-dir\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.716074 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.716038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-dshm\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.719532 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.719486    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328dfdf4-6723-4d54-8902-2d217f6be6a4-tls-certs\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.720162 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.720076    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qcb9\" (UniqueName: \"kubernetes.io/projected/328dfdf4-6723-4d54-8902-2d217f6be6a4-kube-api-access-8qcb9\") pod \"auth-test-model-3616cc-kserve-75b86b89f9-8mrlb\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.785843 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.785795    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:56:53.981236 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:53.981142    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn" podUID="92bf24cf-12a3-4f8c-b6e5-139a91e77a25" containerName="main" containerID="cri-o://5b58cf4bcdd150db64800baf9d69ddcc71ad7fabfd33d20bc73aa34ca5f681f1" gracePeriod=45
Sep 16 07:56:54.055815 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.055791    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"]
Sep 16 07:56:54.056970 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:56:54.056943    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod328dfdf4_6723_4d54_8902_2d217f6be6a4.slice/crio-feba76be649b3f50738c92105ccff74b3e1fef02ec614bf8a83761bb36fa393b WatchSource:0}: Error finding container feba76be649b3f50738c92105ccff74b3e1fef02ec614bf8a83761bb36fa393b: Status 404 returned error can't find the container with id feba76be649b3f50738c92105ccff74b3e1fef02ec614bf8a83761bb36fa393b
Sep 16 07:56:54.209473 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.209448    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:56:54.317989 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.317910    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-model-cache\") pod \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") "
Sep 16 07:56:54.317989 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.317949    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-dshm\") pod \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") "
Sep 16 07:56:54.317989 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.317975    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-tls-certs\") pod \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") "
Sep 16 07:56:54.318225 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.318021    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-tmp-dir\") pod \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") "
Sep 16 07:56:54.318225 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.318058    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmtc6\" (UniqueName: \"kubernetes.io/projected/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-kube-api-access-zmtc6\") pod \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") "
Sep 16 07:56:54.318225 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.318109    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-home\") pod \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\" (UID: \"92bf24cf-12a3-4f8c-b6e5-139a91e77a25\") "
Sep 16 07:56:54.318225 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.318188    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-model-cache" (OuterVolumeSpecName: "model-cache") pod "92bf24cf-12a3-4f8c-b6e5-139a91e77a25" (UID: "92bf24cf-12a3-4f8c-b6e5-139a91e77a25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:56:54.318444 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.318303    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92bf24cf-12a3-4f8c-b6e5-139a91e77a25" (UID: "92bf24cf-12a3-4f8c-b6e5-139a91e77a25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:56:54.318444 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.318424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-home" (OuterVolumeSpecName: "home") pod "92bf24cf-12a3-4f8c-b6e5-139a91e77a25" (UID: "92bf24cf-12a3-4f8c-b6e5-139a91e77a25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:56:54.318554 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.318529    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:56:54.318554 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.318549    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:56:54.318619 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.318562    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:56:54.320114 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.320093    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-dshm" (OuterVolumeSpecName: "dshm") pod "92bf24cf-12a3-4f8c-b6e5-139a91e77a25" (UID: "92bf24cf-12a3-4f8c-b6e5-139a91e77a25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:56:54.320195 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.320113    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92bf24cf-12a3-4f8c-b6e5-139a91e77a25" (UID: "92bf24cf-12a3-4f8c-b6e5-139a91e77a25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 07:56:54.320195 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.320156    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-kube-api-access-zmtc6" (OuterVolumeSpecName: "kube-api-access-zmtc6") pod "92bf24cf-12a3-4f8c-b6e5-139a91e77a25" (UID: "92bf24cf-12a3-4f8c-b6e5-139a91e77a25"). InnerVolumeSpecName "kube-api-access-zmtc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:56:54.419245 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.419219    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmtc6\" (UniqueName: \"kubernetes.io/projected/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-kube-api-access-zmtc6\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:56:54.419245 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.419244    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:56:54.419391 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.419257    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92bf24cf-12a3-4f8c-b6e5-139a91e77a25-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:56:54.874596 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.874559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb" event={"ID":"328dfdf4-6723-4d54-8902-2d217f6be6a4","Type":"ContainerStarted","Data":"c6c9d0d80c975f01d5f983589aabc21a66640200b43144b3faac568e21d46f32"}
Sep 16 07:56:54.875030 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.874608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb" event={"ID":"328dfdf4-6723-4d54-8902-2d217f6be6a4","Type":"ContainerStarted","Data":"feba76be649b3f50738c92105ccff74b3e1fef02ec614bf8a83761bb36fa393b"}
Sep 16 07:56:54.876048 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.876016    2581 generic.go:356] "Generic (PLEG): container finished" podID="92bf24cf-12a3-4f8c-b6e5-139a91e77a25" containerID="5b58cf4bcdd150db64800baf9d69ddcc71ad7fabfd33d20bc73aa34ca5f681f1" exitCode=0
Sep 16 07:56:54.876199 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.876052    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"
Sep 16 07:56:54.876199 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.876102    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn" event={"ID":"92bf24cf-12a3-4f8c-b6e5-139a91e77a25","Type":"ContainerDied","Data":"5b58cf4bcdd150db64800baf9d69ddcc71ad7fabfd33d20bc73aa34ca5f681f1"}
Sep 16 07:56:54.876199 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.876140    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn" event={"ID":"92bf24cf-12a3-4f8c-b6e5-139a91e77a25","Type":"ContainerDied","Data":"de1fb65c365915aedb9c6dd975ba31d417c8ac647c7654d2784f67ee40fa959b"}
Sep 16 07:56:54.876199 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.876160    2581 scope.go:117] "RemoveContainer" containerID="5b58cf4bcdd150db64800baf9d69ddcc71ad7fabfd33d20bc73aa34ca5f681f1"
Sep 16 07:56:54.885727 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.885710    2581 scope.go:117] "RemoveContainer" containerID="5b58cf4bcdd150db64800baf9d69ddcc71ad7fabfd33d20bc73aa34ca5f681f1"
Sep 16 07:56:54.886052 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:56:54.886034    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b58cf4bcdd150db64800baf9d69ddcc71ad7fabfd33d20bc73aa34ca5f681f1\": container with ID starting with 5b58cf4bcdd150db64800baf9d69ddcc71ad7fabfd33d20bc73aa34ca5f681f1 not found: ID does not exist" containerID="5b58cf4bcdd150db64800baf9d69ddcc71ad7fabfd33d20bc73aa34ca5f681f1"
Sep 16 07:56:54.886125 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.886061    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b58cf4bcdd150db64800baf9d69ddcc71ad7fabfd33d20bc73aa34ca5f681f1"} err="failed to get container status \"5b58cf4bcdd150db64800baf9d69ddcc71ad7fabfd33d20bc73aa34ca5f681f1\": rpc error: code = NotFound desc = could not find container \"5b58cf4bcdd150db64800baf9d69ddcc71ad7fabfd33d20bc73aa34ca5f681f1\": container with ID starting with 5b58cf4bcdd150db64800baf9d69ddcc71ad7fabfd33d20bc73aa34ca5f681f1 not found: ID does not exist"
Sep 16 07:56:54.891642 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.891599    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb" podStartSLOduration=1.891586829 podStartE2EDuration="1.891586829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:56:54.890995557 +0000 UTC m=+1375.817607712" watchObservedRunningTime="2026-09-16 07:56:54.891586829 +0000 UTC m=+1375.818198963"
Sep 16 07:56:54.904336 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.904311    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"]
Sep 16 07:56:54.907781 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:54.907759    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/e2e-conflict-a-8a3fd4d9-kserve-659df648f8-mxbxn"]
Sep 16 07:56:55.684215 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:56:55.684184    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92bf24cf-12a3-4f8c-b6e5-139a91e77a25" path="/var/lib/kubelet/pods/92bf24cf-12a3-4f8c-b6e5-139a91e77a25/volumes"
Sep 16 07:57:03.785939 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:03.785906    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:57:03.785939 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:03.785946    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:57:03.798511 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:03.798479    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:57:03.938870 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:03.938845    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 07:57:31.965289 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:31.965254    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"]
Sep 16 07:57:31.965926 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:31.965909    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92bf24cf-12a3-4f8c-b6e5-139a91e77a25" containerName="main"
Sep 16 07:57:31.965926 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:31.965927    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="92bf24cf-12a3-4f8c-b6e5-139a91e77a25" containerName="main"
Sep 16 07:57:31.966018 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:31.965997    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="92bf24cf-12a3-4f8c-b6e5-139a91e77a25" containerName="main"
Sep 16 07:57:31.969080 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:31.969061    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:31.971410 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:31.971388    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"kube-root-ca.crt\""
Sep 16 07:57:31.972439 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:31.972418    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"openshift-service-ca.crt\""
Sep 16 07:57:31.972553 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:31.972428    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"distinct-2-w4-fc27a5-kserve-self-signed-certs\""
Sep 16 07:57:31.972553 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:31.972476    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"default-dockercfg-dkpl6\""
Sep 16 07:57:31.977615 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:31.977592    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"]
Sep 16 07:57:32.092781 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.092749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-home\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.092977 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.092803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-model-cache\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.092977 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.092909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-tmp-dir\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.092977 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.092949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-dshm\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.093091 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.093019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06b891c9-f64a-4833-8bd0-df3d3482f956-tls-certs\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.093091 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.093066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvx6g\" (UniqueName: \"kubernetes.io/projected/06b891c9-f64a-4833-8bd0-df3d3482f956-kube-api-access-lvx6g\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.193842 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.193787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-home\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.194011 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.193862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-model-cache\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.194011 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.193908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-tmp-dir\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.194098 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.194039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-dshm\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.194162 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.194143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06b891c9-f64a-4833-8bd0-df3d3482f956-tls-certs\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.194205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.194167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-home\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.194246 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.194210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvx6g\" (UniqueName: \"kubernetes.io/projected/06b891c9-f64a-4833-8bd0-df3d3482f956-kube-api-access-lvx6g\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.194246 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.194222    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-tmp-dir\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.194343 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.194274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-model-cache\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.196246 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.196220    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-dshm\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.196410 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.196395    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06b891c9-f64a-4833-8bd0-df3d3482f956-tls-certs\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.201954 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.201931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvx6g\" (UniqueName: \"kubernetes.io/projected/06b891c9-f64a-4833-8bd0-df3d3482f956-kube-api-access-lvx6g\") pod \"distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") " pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.281179 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.281091    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:32.555473 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:32.555443    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"]
Sep 16 07:57:32.556685 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:57:32.556656    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06b891c9_f64a_4833_8bd0_df3d3482f956.slice/crio-ac0a07057ff9aea9224bc2135bc28422f654eb176243c63d296e224d774f3f5d WatchSource:0}: Error finding container ac0a07057ff9aea9224bc2135bc28422f654eb176243c63d296e224d774f3f5d: Status 404 returned error can't find the container with id ac0a07057ff9aea9224bc2135bc28422f654eb176243c63d296e224d774f3f5d
Sep 16 07:57:33.088900 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:33.088853    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58" event={"ID":"06b891c9-f64a-4833-8bd0-df3d3482f956","Type":"ContainerStarted","Data":"151f9f63a1558fdb5fff15e1a2f8c6c4d710ba79906e44e6b41653dcbf3049f3"}
Sep 16 07:57:33.088900 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:33.088905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58" event={"ID":"06b891c9-f64a-4833-8bd0-df3d3482f956","Type":"ContainerStarted","Data":"ac0a07057ff9aea9224bc2135bc28422f654eb176243c63d296e224d774f3f5d"}
Sep 16 07:57:33.109311 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:33.109264    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58" podStartSLOduration=2.109246585 podStartE2EDuration="2.109246585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:57:33.10778265 +0000 UTC m=+1414.034394804" watchObservedRunningTime="2026-09-16 07:57:33.109246585 +0000 UTC m=+1414.035858738"
Sep 16 07:57:37.285250 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:37.285210    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"]
Sep 16 07:57:37.285834 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:37.285490    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6" podUID="b091dc61-ca45-4d41-8c11-4b08d23eff42" containerName="istio-proxy" containerID="cri-o://0232567d564ba7a5505aace1af649b5203a45cc15e8bd4dfddd8efaa172ab13c" gracePeriod=30
Sep 16 07:57:42.281581 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.281534    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:42.282091 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.281599    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:42.295021 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.294984    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:42.542352 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.542287    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:57:42.611085 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611046    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-workload-socket\") pod \"b091dc61-ca45-4d41-8c11-4b08d23eff42\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") "
Sep 16 07:57:42.611286 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611118    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-credential-socket\") pod \"b091dc61-ca45-4d41-8c11-4b08d23eff42\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") "
Sep 16 07:57:42.611286 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611150    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-envoy\") pod \"b091dc61-ca45-4d41-8c11-4b08d23eff42\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") "
Sep 16 07:57:42.611286 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611194    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt448\" (UniqueName: \"kubernetes.io/projected/b091dc61-ca45-4d41-8c11-4b08d23eff42-kube-api-access-tt448\") pod \"b091dc61-ca45-4d41-8c11-4b08d23eff42\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") "
Sep 16 07:57:42.611286 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611227    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-workload-certs\") pod \"b091dc61-ca45-4d41-8c11-4b08d23eff42\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") "
Sep 16 07:57:42.611286 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611275    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b091dc61-ca45-4d41-8c11-4b08d23eff42-istiod-ca-cert\") pod \"b091dc61-ca45-4d41-8c11-4b08d23eff42\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") "
Sep 16 07:57:42.611640 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611385    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-podinfo\") pod \"b091dc61-ca45-4d41-8c11-4b08d23eff42\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") "
Sep 16 07:57:42.611640 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611389    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b091dc61-ca45-4d41-8c11-4b08d23eff42" (UID: "b091dc61-ca45-4d41-8c11-4b08d23eff42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:57:42.611640 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-data\") pod \"b091dc61-ca45-4d41-8c11-4b08d23eff42\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") "
Sep 16 07:57:42.611640 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611462    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b091dc61-ca45-4d41-8c11-4b08d23eff42" (UID: "b091dc61-ca45-4d41-8c11-4b08d23eff42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:57:42.611640 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611527    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-token\") pod \"b091dc61-ca45-4d41-8c11-4b08d23eff42\" (UID: \"b091dc61-ca45-4d41-8c11-4b08d23eff42\") "
Sep 16 07:57:42.611944 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611687    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-data" (OuterVolumeSpecName: "istio-data") pod "b091dc61-ca45-4d41-8c11-4b08d23eff42" (UID: "b091dc61-ca45-4d41-8c11-4b08d23eff42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:57:42.612007 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611937    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b091dc61-ca45-4d41-8c11-4b08d23eff42-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b091dc61-ca45-4d41-8c11-4b08d23eff42" (UID: "b091dc61-ca45-4d41-8c11-4b08d23eff42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:57:42.612007 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611949    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:57:42.612007 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611971    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:57:42.612007 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.611986    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:57:42.612224 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.612044    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b091dc61-ca45-4d41-8c11-4b08d23eff42" (UID: "b091dc61-ca45-4d41-8c11-4b08d23eff42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:57:42.613665 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.613640    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b091dc61-ca45-4d41-8c11-4b08d23eff42" (UID: "b091dc61-ca45-4d41-8c11-4b08d23eff42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 07:57:42.613780 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.613726    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b091dc61-ca45-4d41-8c11-4b08d23eff42-kube-api-access-tt448" (OuterVolumeSpecName: "kube-api-access-tt448") pod "b091dc61-ca45-4d41-8c11-4b08d23eff42" (UID: "b091dc61-ca45-4d41-8c11-4b08d23eff42"). InnerVolumeSpecName "kube-api-access-tt448". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:57:42.613945 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.613918    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b091dc61-ca45-4d41-8c11-4b08d23eff42" (UID: "b091dc61-ca45-4d41-8c11-4b08d23eff42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 07:57:42.614035 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.614018    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-token" (OuterVolumeSpecName: "istio-token") pod "b091dc61-ca45-4d41-8c11-4b08d23eff42" (UID: "b091dc61-ca45-4d41-8c11-4b08d23eff42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:57:42.713526 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.713485    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt448\" (UniqueName: \"kubernetes.io/projected/b091dc61-ca45-4d41-8c11-4b08d23eff42-kube-api-access-tt448\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:57:42.713526 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.713520    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:57:42.713526 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.713530    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b091dc61-ca45-4d41-8c11-4b08d23eff42-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:57:42.713526 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.713537    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:57:42.713863 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.713545    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:57:42.713863 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:42.713554    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b091dc61-ca45-4d41-8c11-4b08d23eff42-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:57:43.145490 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:43.145454    2581 generic.go:356] "Generic (PLEG): container finished" podID="b091dc61-ca45-4d41-8c11-4b08d23eff42" containerID="0232567d564ba7a5505aace1af649b5203a45cc15e8bd4dfddd8efaa172ab13c" exitCode=0
Sep 16 07:57:43.145700 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:43.145519    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"
Sep 16 07:57:43.145700 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:43.145532    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6" event={"ID":"b091dc61-ca45-4d41-8c11-4b08d23eff42","Type":"ContainerDied","Data":"0232567d564ba7a5505aace1af649b5203a45cc15e8bd4dfddd8efaa172ab13c"}
Sep 16 07:57:43.145700 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:43.145583    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6" event={"ID":"b091dc61-ca45-4d41-8c11-4b08d23eff42","Type":"ContainerDied","Data":"cf84687738a2232d987c83c67090c17cbd507975461b072b4be1eb38e14c1521"}
Sep 16 07:57:43.145700 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:43.145605    2581 scope.go:117] "RemoveContainer" containerID="0232567d564ba7a5505aace1af649b5203a45cc15e8bd4dfddd8efaa172ab13c"
Sep 16 07:57:43.156335 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:43.156316    2581 scope.go:117] "RemoveContainer" containerID="8e9e949f6af38dbea7a35f2d7d610529a0f28fcde5e4d35da81a099e784ce828"
Sep 16 07:57:43.158400 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:43.158380    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 07:57:43.166092 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:43.166065    2581 scope.go:117] "RemoveContainer" containerID="0232567d564ba7a5505aace1af649b5203a45cc15e8bd4dfddd8efaa172ab13c"
Sep 16 07:57:43.166899 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:57:43.166870    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0232567d564ba7a5505aace1af649b5203a45cc15e8bd4dfddd8efaa172ab13c\": container with ID starting with 0232567d564ba7a5505aace1af649b5203a45cc15e8bd4dfddd8efaa172ab13c not found: ID does not exist" containerID="0232567d564ba7a5505aace1af649b5203a45cc15e8bd4dfddd8efaa172ab13c"
Sep 16 07:57:43.167022 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:43.166911    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0232567d564ba7a5505aace1af649b5203a45cc15e8bd4dfddd8efaa172ab13c"} err="failed to get container status \"0232567d564ba7a5505aace1af649b5203a45cc15e8bd4dfddd8efaa172ab13c\": rpc error: code = NotFound desc = could not find container \"0232567d564ba7a5505aace1af649b5203a45cc15e8bd4dfddd8efaa172ab13c\": container with ID starting with 0232567d564ba7a5505aace1af649b5203a45cc15e8bd4dfddd8efaa172ab13c not found: ID does not exist"
Sep 16 07:57:43.167022 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:43.166938    2581 scope.go:117] "RemoveContainer" containerID="8e9e949f6af38dbea7a35f2d7d610529a0f28fcde5e4d35da81a099e784ce828"
Sep 16 07:57:43.167284 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:57:43.167262    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e9e949f6af38dbea7a35f2d7d610529a0f28fcde5e4d35da81a099e784ce828\": container with ID starting with 8e9e949f6af38dbea7a35f2d7d610529a0f28fcde5e4d35da81a099e784ce828 not found: ID does not exist" containerID="8e9e949f6af38dbea7a35f2d7d610529a0f28fcde5e4d35da81a099e784ce828"
Sep 16 07:57:43.167333 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:43.167290    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9e949f6af38dbea7a35f2d7d610529a0f28fcde5e4d35da81a099e784ce828"} err="failed to get container status \"8e9e949f6af38dbea7a35f2d7d610529a0f28fcde5e4d35da81a099e784ce828\": rpc error: code = NotFound desc = could not find container \"8e9e949f6af38dbea7a35f2d7d610529a0f28fcde5e4d35da81a099e784ce828\": container with ID starting with 8e9e949f6af38dbea7a35f2d7d610529a0f28fcde5e4d35da81a099e784ce828 not found: ID does not exist"
Sep 16 07:57:43.168913 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:43.168891    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"]
Sep 16 07:57:43.173911 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:43.173881    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fc2a294d-openshift-default-8597ff85fb-m9ws6"]
Sep 16 07:57:43.684757 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:43.684725    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b091dc61-ca45-4d41-8c11-4b08d23eff42" path="/var/lib/kubelet/pods/b091dc61-ca45-4d41-8c11-4b08d23eff42/volumes"
Sep 16 07:57:47.191320 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.191286    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"]
Sep 16 07:57:47.192017 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.191994    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b091dc61-ca45-4d41-8c11-4b08d23eff42" containerName="istio-proxy"
Sep 16 07:57:47.192141 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.192019    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b091dc61-ca45-4d41-8c11-4b08d23eff42" containerName="istio-proxy"
Sep 16 07:57:47.192141 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.192032    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b091dc61-ca45-4d41-8c11-4b08d23eff42" containerName="istio-proxy"
Sep 16 07:57:47.192141 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.192047    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b091dc61-ca45-4d41-8c11-4b08d23eff42" containerName="istio-proxy"
Sep 16 07:57:47.192293 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.192169    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b091dc61-ca45-4d41-8c11-4b08d23eff42" containerName="istio-proxy"
Sep 16 07:57:47.192293 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.192186    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b091dc61-ca45-4d41-8c11-4b08d23eff42" containerName="istio-proxy"
Sep 16 07:57:47.197006 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.196970    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.200005 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.199948    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e9fc8c\"/\"openshift-service-ca.crt\""
Sep 16 07:57:47.201114 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.201088    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e9fc8c\"/\"default-dockercfg-bd6xl\""
Sep 16 07:57:47.201229 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.201088    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e9fc8c\"/\"kube-root-ca.crt\""
Sep 16 07:57:47.201398 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.201379    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e9fc8c\"/\"auth-test-model-e9fc8c-kserve-self-signed-certs\""
Sep 16 07:57:47.212482 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.212451    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"]
Sep 16 07:57:47.259720 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.259685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-model-cache\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.259720 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.259720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-tls-certs\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.259990 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.259756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-dshm\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.259990 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.259805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-tmp-dir\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.259990 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.259881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z24md\" (UniqueName: \"kubernetes.io/projected/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-kube-api-access-z24md\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.259990 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.259917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-home\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.360810 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.360775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-dshm\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.361022 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.360870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-tmp-dir\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.361022 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.360927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z24md\" (UniqueName: \"kubernetes.io/projected/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-kube-api-access-z24md\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.361022 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.360980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-home\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.361205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.361066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-model-cache\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.361205 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.361096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-tls-certs\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.361310 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.361251    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-tmp-dir\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.361310 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.361293    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-home\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.361453 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.361428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-model-cache\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.363041 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.363021    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-dshm\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.363392 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.363373    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-tls-certs\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.369560 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.369539    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z24md\" (UniqueName: \"kubernetes.io/projected/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-kube-api-access-z24md\") pod \"auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.507129 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.507038    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:47.778641 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:47.778533    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"]
Sep 16 07:57:47.780756 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:57:47.780730    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc42595f4_91f8_45ff_9a0e_3ca5b75f6ae2.slice/crio-1515516abb6a2c282f489c89bb5c5a4408338e64ba383a43f538a6e447dcc97d WatchSource:0}: Error finding container 1515516abb6a2c282f489c89bb5c5a4408338e64ba383a43f538a6e447dcc97d: Status 404 returned error can't find the container with id 1515516abb6a2c282f489c89bb5c5a4408338e64ba383a43f538a6e447dcc97d
Sep 16 07:57:48.174698 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:48.174666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf" event={"ID":"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2","Type":"ContainerStarted","Data":"287b7c2f77175dffd8b7aa16a5ab7713040d115c4ec0194019f4bc9d9a727625"}
Sep 16 07:57:48.174877 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:48.174708    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf" event={"ID":"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2","Type":"ContainerStarted","Data":"1515516abb6a2c282f489c89bb5c5a4408338e64ba383a43f538a6e447dcc97d"}
Sep 16 07:57:48.191853 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:48.191792    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf" podStartSLOduration=1.191776444 podStartE2EDuration="1.191776444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:57:48.190155603 +0000 UTC m=+1429.116767755" watchObservedRunningTime="2026-09-16 07:57:48.191776444 +0000 UTC m=+1429.118388597"
Sep 16 07:57:57.507499 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:57.507421    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:57.507499 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:57.507471    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:57.520495 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:57.520471    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:58.241149 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:58.241119    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 07:57:59.804093 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:57:59.804064    2581 scope.go:117] "RemoveContainer" containerID="3fb106a3b6d86215ab344d71beb4ca4e2b6529b3c498b31e8dbe09290f620a4c"
Sep 16 07:58:31.520606 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.520567    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"]
Sep 16 07:58:31.524552 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.524530    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.527241 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.527214    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-987ea6-openshift-default-dockercfg-c2vrr\""
Sep 16 07:58:31.537302 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.537278    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"]
Sep 16 07:58:31.581131 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.581089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-istio-envoy\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.581264 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.581139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-istio-data\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.581264 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.581198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0fdf27-753f-4005-847b-68c869d325ae-istio-token\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.581264 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.581233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-workload-socket\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.581372 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.581324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0fdf27-753f-4005-847b-68c869d325ae-istiod-ca-cert\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.581409 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.581386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-workload-certs\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.581444 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.581409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0fdf27-753f-4005-847b-68c869d325ae-istio-podinfo\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.581444 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.581437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-credential-socket\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.581509 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.581484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2dcs\" (UniqueName: \"kubernetes.io/projected/ca0fdf27-753f-4005-847b-68c869d325ae-kube-api-access-t2dcs\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.682790 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.682743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0fdf27-753f-4005-847b-68c869d325ae-istiod-ca-cert\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.683003 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.682894    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-workload-certs\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.683003 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.682924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0fdf27-753f-4005-847b-68c869d325ae-istio-podinfo\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.683003 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.682950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-credential-socket\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.683180 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.683005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2dcs\" (UniqueName: \"kubernetes.io/projected/ca0fdf27-753f-4005-847b-68c869d325ae-kube-api-access-t2dcs\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.683180 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.683046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-istio-envoy\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.683180 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.683075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-istio-data\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.683180 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.683115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0fdf27-753f-4005-847b-68c869d325ae-istio-token\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.683180 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.683147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-workload-socket\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.683444 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.683368    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-credential-socket\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.683557 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.683531    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-workload-socket\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.683705 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.683568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0fdf27-753f-4005-847b-68c869d325ae-istiod-ca-cert\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.683902 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.683877    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-workload-certs\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.684059 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.683902    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-istio-data\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.685631 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.685610    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0fdf27-753f-4005-847b-68c869d325ae-istio-podinfo\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.685912 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.685888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-istio-envoy\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.691217 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.691192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2dcs\" (UniqueName: \"kubernetes.io/projected/ca0fdf27-753f-4005-847b-68c869d325ae-kube-api-access-t2dcs\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.691327 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.691306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0fdf27-753f-4005-847b-68c869d325ae-istio-token\") pod \"e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") " pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:31.838606 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:31.838530    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:32.123063 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:32.123038    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"]
Sep 16 07:58:32.124443 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:58:32.124415    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0fdf27_753f_4005_847b_68c869d325ae.slice/crio-2841a3bdb20427ef6cdf70558343ac31ddca01a6da8147eccdce19ba966ebdf4 WatchSource:0}: Error finding container 2841a3bdb20427ef6cdf70558343ac31ddca01a6da8147eccdce19ba966ebdf4: Status 404 returned error can't find the container with id 2841a3bdb20427ef6cdf70558343ac31ddca01a6da8147eccdce19ba966ebdf4
Sep 16 07:58:32.126834 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:32.126789    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:58:32.126940 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:32.126893    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:58:32.127005 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:32.126937    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:58:32.421947 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:32.421908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk" event={"ID":"ca0fdf27-753f-4005-847b-68c869d325ae","Type":"ContainerStarted","Data":"26602db4ca79043b7818384b565149ef4474f358e9dae9f2a287b62c0adda2ef"}
Sep 16 07:58:32.422102 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:32.421970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk" event={"ID":"ca0fdf27-753f-4005-847b-68c869d325ae","Type":"ContainerStarted","Data":"2841a3bdb20427ef6cdf70558343ac31ddca01a6da8147eccdce19ba966ebdf4"}
Sep 16 07:58:32.440527 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:32.440482    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk" podStartSLOduration=1.440466701 podStartE2EDuration="1.440466701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:58:32.439696107 +0000 UTC m=+1473.366308273" watchObservedRunningTime="2026-09-16 07:58:32.440466701 +0000 UTC m=+1473.367078853"
Sep 16 07:58:32.839513 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:32.839441    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:33.845066 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:33.845037    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:34.433784 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:34.433752    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:34.434775 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:34.434752    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:58:39.468458 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.468428    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"]
Sep 16 07:58:39.472430 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.472412    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"
Sep 16 07:58:39.474778 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.474753    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-987ea6\""
Sep 16 07:58:39.474902 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.474814    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-987ea6-dockercfg-nkbnx\""
Sep 16 07:58:39.482341 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.482318    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"]
Sep 16 07:58:39.514137 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.514110    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"]
Sep 16 07:58:39.517530 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.517512    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"
Sep 16 07:58:39.526887 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.526852    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"]
Sep 16 07:58:39.560719 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.560693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/742901d8-bc38-432f-b3bd-8ad9e10a7ed7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6\" (UID: \"742901d8-bc38-432f-b3bd-8ad9e10a7ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"
Sep 16 07:58:39.560812 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.560722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxvv7\" (UniqueName: \"kubernetes.io/projected/742901d8-bc38-432f-b3bd-8ad9e10a7ed7-kube-api-access-lxvv7\") pod \"payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6\" (UID: \"742901d8-bc38-432f-b3bd-8ad9e10a7ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"
Sep 16 07:58:39.661591 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.661546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67dcac13-2152-4225-bfd1-782cd53568d1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-987ea6-844c556647-rg5ns\" (UID: \"67dcac13-2152-4225-bfd1-782cd53568d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"
Sep 16 07:58:39.661591 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.661600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjmvg\" (UniqueName: \"kubernetes.io/projected/67dcac13-2152-4225-bfd1-782cd53568d1-kube-api-access-jjmvg\") pod \"payload-processing-e2e-api-a-987ea6-844c556647-rg5ns\" (UID: \"67dcac13-2152-4225-bfd1-782cd53568d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"
Sep 16 07:58:39.661801 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.661687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/742901d8-bc38-432f-b3bd-8ad9e10a7ed7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6\" (UID: \"742901d8-bc38-432f-b3bd-8ad9e10a7ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"
Sep 16 07:58:39.661801 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.661718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxvv7\" (UniqueName: \"kubernetes.io/projected/742901d8-bc38-432f-b3bd-8ad9e10a7ed7-kube-api-access-lxvv7\") pod \"payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6\" (UID: \"742901d8-bc38-432f-b3bd-8ad9e10a7ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"
Sep 16 07:58:39.662342 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.662324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/742901d8-bc38-432f-b3bd-8ad9e10a7ed7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6\" (UID: \"742901d8-bc38-432f-b3bd-8ad9e10a7ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"
Sep 16 07:58:39.669539 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.669518    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxvv7\" (UniqueName: \"kubernetes.io/projected/742901d8-bc38-432f-b3bd-8ad9e10a7ed7-kube-api-access-lxvv7\") pod \"payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6\" (UID: \"742901d8-bc38-432f-b3bd-8ad9e10a7ed7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"
Sep 16 07:58:39.762476 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.762417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67dcac13-2152-4225-bfd1-782cd53568d1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-987ea6-844c556647-rg5ns\" (UID: \"67dcac13-2152-4225-bfd1-782cd53568d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"
Sep 16 07:58:39.762476 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.762452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjmvg\" (UniqueName: \"kubernetes.io/projected/67dcac13-2152-4225-bfd1-782cd53568d1-kube-api-access-jjmvg\") pod \"payload-processing-e2e-api-a-987ea6-844c556647-rg5ns\" (UID: \"67dcac13-2152-4225-bfd1-782cd53568d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"
Sep 16 07:58:39.763129 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.763112    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67dcac13-2152-4225-bfd1-782cd53568d1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-987ea6-844c556647-rg5ns\" (UID: \"67dcac13-2152-4225-bfd1-782cd53568d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"
Sep 16 07:58:39.770204 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.770180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjmvg\" (UniqueName: \"kubernetes.io/projected/67dcac13-2152-4225-bfd1-782cd53568d1-kube-api-access-jjmvg\") pod \"payload-processing-e2e-api-a-987ea6-844c556647-rg5ns\" (UID: \"67dcac13-2152-4225-bfd1-782cd53568d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"
Sep 16 07:58:39.783890 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.783871    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"
Sep 16 07:58:39.828883 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:39.828857    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"
Sep 16 07:58:40.055698 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:40.055669    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"]
Sep 16 07:58:40.056935 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:58:40.056902    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742901d8_bc38_432f_b3bd_8ad9e10a7ed7.slice/crio-018c1988c33de846925c9fa835298d5af030aa0614ab5b13244f042c12a44ad6 WatchSource:0}: Error finding container 018c1988c33de846925c9fa835298d5af030aa0614ab5b13244f042c12a44ad6: Status 404 returned error can't find the container with id 018c1988c33de846925c9fa835298d5af030aa0614ab5b13244f042c12a44ad6
Sep 16 07:58:40.114956 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:40.114931    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"]
Sep 16 07:58:40.115042 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:58:40.114931    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67dcac13_2152_4225_bfd1_782cd53568d1.slice/crio-c450877bff02a2405104cc7da7c0dc6969d4a109d4edf34202c6b2a616ff92f7 WatchSource:0}: Error finding container c450877bff02a2405104cc7da7c0dc6969d4a109d4edf34202c6b2a616ff92f7: Status 404 returned error can't find the container with id c450877bff02a2405104cc7da7c0dc6969d4a109d4edf34202c6b2a616ff92f7
Sep 16 07:58:40.469092 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:40.469055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns" event={"ID":"67dcac13-2152-4225-bfd1-782cd53568d1","Type":"ContainerStarted","Data":"ebc528de1865bde9257305c80f47a66f3faef7bb15f8ec9a9323fcf83c00c4dc"}
Sep 16 07:58:40.469553 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:40.469122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns" event={"ID":"67dcac13-2152-4225-bfd1-782cd53568d1","Type":"ContainerStarted","Data":"c450877bff02a2405104cc7da7c0dc6969d4a109d4edf34202c6b2a616ff92f7"}
Sep 16 07:58:40.469553 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:40.469194    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"
Sep 16 07:58:40.470696 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:40.470673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6" event={"ID":"742901d8-bc38-432f-b3bd-8ad9e10a7ed7","Type":"ContainerStarted","Data":"aef7c1c4c61388db0f972416e49b209c7b303de3d154c376834428e4da2ba0e5"}
Sep 16 07:58:40.470839 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:40.470708    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6" event={"ID":"742901d8-bc38-432f-b3bd-8ad9e10a7ed7","Type":"ContainerStarted","Data":"018c1988c33de846925c9fa835298d5af030aa0614ab5b13244f042c12a44ad6"}
Sep 16 07:58:40.470839 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:40.470798    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"
Sep 16 07:58:40.486118 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:40.486064    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns" podStartSLOduration=1.486051913 podStartE2EDuration="1.486051913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:58:40.484568219 +0000 UTC m=+1481.411180371" watchObservedRunningTime="2026-09-16 07:58:40.486051913 +0000 UTC m=+1481.412664064"
Sep 16 07:58:40.500216 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:40.500173    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6" podStartSLOduration=1.500160323 podStartE2EDuration="1.500160323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:58:40.498873302 +0000 UTC m=+1481.425485454" watchObservedRunningTime="2026-09-16 07:58:40.500160323 +0000 UTC m=+1481.426772475"
Sep 16 07:58:51.477864 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:51.477815    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"
Sep 16 07:58:51.478276 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:51.477983    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"
Sep 16 07:58:59.628604 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:59.628578    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 07:58:59.631590 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:58:59.631563    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 07:59:05.207621 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.207587    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"]
Sep 16 07:59:05.212350 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.212330    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.215139 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.215115    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3a5b4c-openshift-default-dockercfg-sbvnn\""
Sep 16 07:59:05.223576 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.223556    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"]
Sep 16 07:59:05.307626 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.307583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-credential-socket\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.307963 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.307938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-token\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.308176 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.308158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-workload-socket\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.308323 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.308306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5qz6\" (UniqueName: \"kubernetes.io/projected/d391f1c7-023e-485b-8709-7f0e4336eafc-kube-api-access-g5qz6\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.308488 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.308473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-envoy\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.308644 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.308629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-workload-certs\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.308776 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.308761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-data\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.308975 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.308957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d391f1c7-023e-485b-8709-7f0e4336eafc-istiod-ca-cert\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.309133 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.309118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-podinfo\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.411026 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.410990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d391f1c7-023e-485b-8709-7f0e4336eafc-istiod-ca-cert\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.411228 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.411050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-podinfo\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.411228 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.411090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-credential-socket\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.411228 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.411125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-token\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.411228 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.411182    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-workload-socket\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.411228 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.411215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5qz6\" (UniqueName: \"kubernetes.io/projected/d391f1c7-023e-485b-8709-7f0e4336eafc-kube-api-access-g5qz6\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.411511 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.411280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-envoy\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.411511 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.411331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-workload-certs\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.411511 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.411356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-data\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.411668 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.411572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-workload-socket\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.411734 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.411670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-data\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.411843 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.411806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-workload-certs\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.411930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.411875    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d391f1c7-023e-485b-8709-7f0e4336eafc-istiod-ca-cert\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.411930 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.411902    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-credential-socket\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.413722 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.413698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-envoy\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.414103 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.414086    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-podinfo\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.422363 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.422338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5qz6\" (UniqueName: \"kubernetes.io/projected/d391f1c7-023e-485b-8709-7f0e4336eafc-kube-api-access-g5qz6\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.423227 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.423204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-token\") pod \"e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") " pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.524064 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.523985    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:05.803025 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.803000    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"]
Sep 16 07:59:05.803917 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:59:05.803891    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd391f1c7_023e_485b_8709_7f0e4336eafc.slice/crio-bf7c19e452d391b9f15316affa2f97adfca92a8991eec2a4279b72b6eba6309d WatchSource:0}: Error finding container bf7c19e452d391b9f15316affa2f97adfca92a8991eec2a4279b72b6eba6309d: Status 404 returned error can't find the container with id bf7c19e452d391b9f15316affa2f97adfca92a8991eec2a4279b72b6eba6309d
Sep 16 07:59:05.806567 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.806529    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:59:05.806639 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.806599    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:59:05.806639 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:05.806628    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:59:06.619644 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:06.619609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8" event={"ID":"d391f1c7-023e-485b-8709-7f0e4336eafc","Type":"ContainerStarted","Data":"848517fec7ab75f1bc63f6b21de3f5569a0b11e7fa88fc3a3ef785f988a8f554"}
Sep 16 07:59:06.620071 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:06.619670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8" event={"ID":"d391f1c7-023e-485b-8709-7f0e4336eafc","Type":"ContainerStarted","Data":"bf7c19e452d391b9f15316affa2f97adfca92a8991eec2a4279b72b6eba6309d"}
Sep 16 07:59:06.645508 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:06.645460    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8" podStartSLOduration=1.645444897 podStartE2EDuration="1.645444897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:59:06.643651294 +0000 UTC m=+1507.570263445" watchObservedRunningTime="2026-09-16 07:59:06.645444897 +0000 UTC m=+1507.572057048"
Sep 16 07:59:07.524601 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:07.524571    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:07.529574 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:07.529548    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:07.626302 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:07.626275    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:07.627099 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:07.627080    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:41.438353 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:41.438314    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2"]
Sep 16 07:59:41.442745 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:41.442724    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2"
Sep 16 07:59:41.445530 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:41.445322    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 07:59:41.445666 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:41.445562    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\""
Sep 16 07:59:41.446810 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:41.446782    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2"]
Sep 16 07:59:41.497062 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:41.497034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e5dfc34-0ed6-4de6-846b-51992e01f54a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2\" (UID: \"1e5dfc34-0ed6-4de6-846b-51992e01f54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2"
Sep 16 07:59:41.598319 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:41.598285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e5dfc34-0ed6-4de6-846b-51992e01f54a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2\" (UID: \"1e5dfc34-0ed6-4de6-846b-51992e01f54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2"
Sep 16 07:59:41.599001 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:41.598975    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e5dfc34-0ed6-4de6-846b-51992e01f54a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2\" (UID: \"1e5dfc34-0ed6-4de6-846b-51992e01f54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2"
Sep 16 07:59:41.759707 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:41.759613    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2"
Sep 16 07:59:42.042285 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:42.042206    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2"]
Sep 16 07:59:42.044313 ip-10-0-139-143 kubenswrapper[2581]: W0916 07:59:42.044286    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5dfc34_0ed6_4de6_846b_51992e01f54a.slice/crio-0c497e0b3c04772e371e41aeec1134cc9e8d9f7a8afc99a460e2d419e6950345 WatchSource:0}: Error finding container 0c497e0b3c04772e371e41aeec1134cc9e8d9f7a8afc99a460e2d419e6950345: Status 404 returned error can't find the container with id 0c497e0b3c04772e371e41aeec1134cc9e8d9f7a8afc99a460e2d419e6950345
Sep 16 07:59:42.852733 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:42.852685    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2" event={"ID":"1e5dfc34-0ed6-4de6-846b-51992e01f54a","Type":"ContainerStarted","Data":"39440163c4281d3993c8a3de8e1806f3e1606a96616cb5eb5bda134dccfba721"}
Sep 16 07:59:42.853207 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:42.852756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2" event={"ID":"1e5dfc34-0ed6-4de6-846b-51992e01f54a","Type":"ContainerStarted","Data":"0c497e0b3c04772e371e41aeec1134cc9e8d9f7a8afc99a460e2d419e6950345"}
Sep 16 07:59:42.854331 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:42.854312    2581 generic.go:356] "Generic (PLEG): container finished" podID="d391f1c7-023e-485b-8709-7f0e4336eafc" containerID="848517fec7ab75f1bc63f6b21de3f5569a0b11e7fa88fc3a3ef785f988a8f554" exitCode=0
Sep 16 07:59:42.854424 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:42.854351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8" event={"ID":"d391f1c7-023e-485b-8709-7f0e4336eafc","Type":"ContainerDied","Data":"848517fec7ab75f1bc63f6b21de3f5569a0b11e7fa88fc3a3ef785f988a8f554"}
Sep 16 07:59:42.854722 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:42.854708    2581 scope.go:117] "RemoveContainer" containerID="848517fec7ab75f1bc63f6b21de3f5569a0b11e7fa88fc3a3ef785f988a8f554"
Sep 16 07:59:42.855987 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:42.855953    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:59:42.856103 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:42.856044    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:59:42.856103 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:42.856093    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:59:42.867863 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:42.867806    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2" podStartSLOduration=1.867791768 podStartE2EDuration="1.867791768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:59:42.867025786 +0000 UTC m=+1543.793637942" watchObservedRunningTime="2026-09-16 07:59:42.867791768 +0000 UTC m=+1543.794403921"
Sep 16 07:59:43.861288 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:43.861250    2581 generic.go:356] "Generic (PLEG): container finished" podID="1e5dfc34-0ed6-4de6-846b-51992e01f54a" containerID="39440163c4281d3993c8a3de8e1806f3e1606a96616cb5eb5bda134dccfba721" exitCode=0
Sep 16 07:59:43.861733 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:43.861335    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2" event={"ID":"1e5dfc34-0ed6-4de6-846b-51992e01f54a","Type":"ContainerDied","Data":"39440163c4281d3993c8a3de8e1806f3e1606a96616cb5eb5bda134dccfba721"}
Sep 16 07:59:43.863300 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:43.863276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8" event={"ID":"d391f1c7-023e-485b-8709-7f0e4336eafc","Type":"ContainerStarted","Data":"7e9b5d1021a71620431a32070de00a28a3104fefe8b3620f95332de307769e45"}
Sep 16 07:59:44.524912 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:44.524867    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:44.525180 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:44.525149    2581 patch_prober.go:28] interesting pod/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 16 07:59:44.525272 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:44.525218    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8" podUID="d391f1c7-023e-485b-8709-7f0e4336eafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 16 07:59:45.098315 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:45.098290    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2"
Sep 16 07:59:45.168556 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:45.168515    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e5dfc34-0ed6-4de6-846b-51992e01f54a-maas-api-service-ca\") pod \"1e5dfc34-0ed6-4de6-846b-51992e01f54a\" (UID: \"1e5dfc34-0ed6-4de6-846b-51992e01f54a\") "
Sep 16 07:59:45.169074 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:45.169025    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5dfc34-0ed6-4de6-846b-51992e01f54a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e5dfc34-0ed6-4de6-846b-51992e01f54a" (UID: "1e5dfc34-0ed6-4de6-846b-51992e01f54a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:59:45.270286 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:45.270252    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e5dfc34-0ed6-4de6-846b-51992e01f54a-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:59:45.525187 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:45.525092    2581 patch_prober.go:28] interesting pod/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 16 07:59:45.525458 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:45.525412    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8" podUID="d391f1c7-023e-485b-8709-7f0e4336eafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 16 07:59:45.876980 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:45.876900    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2" event={"ID":"1e5dfc34-0ed6-4de6-846b-51992e01f54a","Type":"ContainerDied","Data":"0c497e0b3c04772e371e41aeec1134cc9e8d9f7a8afc99a460e2d419e6950345"}
Sep 16 07:59:45.876980 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:45.876922    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2"
Sep 16 07:59:45.876980 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:45.876950    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c497e0b3c04772e371e41aeec1134cc9e8d9f7a8afc99a460e2d419e6950345"
Sep 16 07:59:46.525146 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:46.525099    2581 patch_prober.go:28] interesting pod/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 16 07:59:46.525643 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:46.525172    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8" podUID="d391f1c7-023e-485b-8709-7f0e4336eafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 16 07:59:47.524841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:47.524568    2581 patch_prober.go:28] interesting pod/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 16 07:59:47.524841 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:47.524629    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8" podUID="d391f1c7-023e-485b-8709-7f0e4336eafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 16 07:59:48.525517 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:48.525048    2581 patch_prober.go:28] interesting pod/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 16 07:59:48.525517 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:48.525138    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8" podUID="d391f1c7-023e-485b-8709-7f0e4336eafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 16 07:59:49.434976 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:49.434939    2581 patch_prober.go:28] interesting pod/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 16 07:59:49.435195 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:49.435009    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk" podUID="ca0fdf27-753f-4005-847b-68c869d325ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 16 07:59:49.525051 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:49.524968    2581 patch_prober.go:28] interesting pod/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 16 07:59:49.525051 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:49.525022    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8" podUID="d391f1c7-023e-485b-8709-7f0e4336eafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 16 07:59:49.903872 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:49.903814    2581 generic.go:356] "Generic (PLEG): container finished" podID="ca0fdf27-753f-4005-847b-68c869d325ae" containerID="26602db4ca79043b7818384b565149ef4474f358e9dae9f2a287b62c0adda2ef" exitCode=0
Sep 16 07:59:49.903872 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:49.903849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk" event={"ID":"ca0fdf27-753f-4005-847b-68c869d325ae","Type":"ContainerDied","Data":"26602db4ca79043b7818384b565149ef4474f358e9dae9f2a287b62c0adda2ef"}
Sep 16 07:59:49.904375 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:49.904359    2581 scope.go:117] "RemoveContainer" containerID="26602db4ca79043b7818384b565149ef4474f358e9dae9f2a287b62c0adda2ef"
Sep 16 07:59:49.905534 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:49.905509    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:59:49.905585 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:49.905575    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:59:49.905622 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:49.905604    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 07:59:50.529464 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:50.529433    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:50.529732 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:50.529713    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:50.530476 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:50.530457    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 07:59:50.911179 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:50.911146    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk" event={"ID":"ca0fdf27-753f-4005-847b-68c869d325ae","Type":"ContainerStarted","Data":"2389c24d01450295bc671e19f63022117a582bbd03f3d8d54c2dcfbf145656b1"}
Sep 16 07:59:51.839687 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:51.839650    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:59:51.843897 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:51.843870    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:59:51.917257 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:51.917225    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:59:51.918156 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:51.918138    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 07:59:58.629811 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.629774    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"]
Sep 16 07:59:58.630235 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.630138    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns" podUID="67dcac13-2152-4225-bfd1-782cd53568d1" containerName="payload-processing" containerID="cri-o://ebc528de1865bde9257305c80f47a66f3faef7bb15f8ec9a9323fcf83c00c4dc" gracePeriod=30
Sep 16 07:59:58.643068 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.643033    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"]
Sep 16 07:59:58.643301 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.643273    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6" podUID="742901d8-bc38-432f-b3bd-8ad9e10a7ed7" containerName="payload-pre-processing" containerID="cri-o://aef7c1c4c61388db0f972416e49b209c7b303de3d154c376834428e4da2ba0e5" gracePeriod=30
Sep 16 07:59:58.813771 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.813430    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2"]
Sep 16 07:59:58.817177 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.817145    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-987ea6-bqhl2"]
Sep 16 07:59:58.931756 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.931729    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"
Sep 16 07:59:58.935114 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.935091    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"
Sep 16 07:59:58.960563 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.960531    2581 generic.go:356] "Generic (PLEG): container finished" podID="67dcac13-2152-4225-bfd1-782cd53568d1" containerID="ebc528de1865bde9257305c80f47a66f3faef7bb15f8ec9a9323fcf83c00c4dc" exitCode=0
Sep 16 07:59:58.960766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.960590    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"
Sep 16 07:59:58.960766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.960608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns" event={"ID":"67dcac13-2152-4225-bfd1-782cd53568d1","Type":"ContainerDied","Data":"ebc528de1865bde9257305c80f47a66f3faef7bb15f8ec9a9323fcf83c00c4dc"}
Sep 16 07:59:58.960766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.960666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns" event={"ID":"67dcac13-2152-4225-bfd1-782cd53568d1","Type":"ContainerDied","Data":"c450877bff02a2405104cc7da7c0dc6969d4a109d4edf34202c6b2a616ff92f7"}
Sep 16 07:59:58.960766 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.960697    2581 scope.go:117] "RemoveContainer" containerID="ebc528de1865bde9257305c80f47a66f3faef7bb15f8ec9a9323fcf83c00c4dc"
Sep 16 07:59:58.962456 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.962428    2581 generic.go:356] "Generic (PLEG): container finished" podID="742901d8-bc38-432f-b3bd-8ad9e10a7ed7" containerID="aef7c1c4c61388db0f972416e49b209c7b303de3d154c376834428e4da2ba0e5" exitCode=0
Sep 16 07:59:58.962581 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.962475    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"
Sep 16 07:59:58.962581 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.962506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6" event={"ID":"742901d8-bc38-432f-b3bd-8ad9e10a7ed7","Type":"ContainerDied","Data":"aef7c1c4c61388db0f972416e49b209c7b303de3d154c376834428e4da2ba0e5"}
Sep 16 07:59:58.962581 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.962562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6" event={"ID":"742901d8-bc38-432f-b3bd-8ad9e10a7ed7","Type":"ContainerDied","Data":"018c1988c33de846925c9fa835298d5af030aa0614ab5b13244f042c12a44ad6"}
Sep 16 07:59:58.970755 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.970666    2581 scope.go:117] "RemoveContainer" containerID="ebc528de1865bde9257305c80f47a66f3faef7bb15f8ec9a9323fcf83c00c4dc"
Sep 16 07:59:58.971230 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:59:58.971203    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc528de1865bde9257305c80f47a66f3faef7bb15f8ec9a9323fcf83c00c4dc\": container with ID starting with ebc528de1865bde9257305c80f47a66f3faef7bb15f8ec9a9323fcf83c00c4dc not found: ID does not exist" containerID="ebc528de1865bde9257305c80f47a66f3faef7bb15f8ec9a9323fcf83c00c4dc"
Sep 16 07:59:58.971358 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.971240    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc528de1865bde9257305c80f47a66f3faef7bb15f8ec9a9323fcf83c00c4dc"} err="failed to get container status \"ebc528de1865bde9257305c80f47a66f3faef7bb15f8ec9a9323fcf83c00c4dc\": rpc error: code = NotFound desc = could not find container \"ebc528de1865bde9257305c80f47a66f3faef7bb15f8ec9a9323fcf83c00c4dc\": container with ID starting with ebc528de1865bde9257305c80f47a66f3faef7bb15f8ec9a9323fcf83c00c4dc not found: ID does not exist"
Sep 16 07:59:58.971358 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:58.971263    2581 scope.go:117] "RemoveContainer" containerID="aef7c1c4c61388db0f972416e49b209c7b303de3d154c376834428e4da2ba0e5"
Sep 16 07:59:59.026844 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.018092    2581 scope.go:117] "RemoveContainer" containerID="aef7c1c4c61388db0f972416e49b209c7b303de3d154c376834428e4da2ba0e5"
Sep 16 07:59:59.026844 ip-10-0-139-143 kubenswrapper[2581]: E0916 07:59:59.018712    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef7c1c4c61388db0f972416e49b209c7b303de3d154c376834428e4da2ba0e5\": container with ID starting with aef7c1c4c61388db0f972416e49b209c7b303de3d154c376834428e4da2ba0e5 not found: ID does not exist" containerID="aef7c1c4c61388db0f972416e49b209c7b303de3d154c376834428e4da2ba0e5"
Sep 16 07:59:59.026844 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.018749    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef7c1c4c61388db0f972416e49b209c7b303de3d154c376834428e4da2ba0e5"} err="failed to get container status \"aef7c1c4c61388db0f972416e49b209c7b303de3d154c376834428e4da2ba0e5\": rpc error: code = NotFound desc = could not find container \"aef7c1c4c61388db0f972416e49b209c7b303de3d154c376834428e4da2ba0e5\": container with ID starting with aef7c1c4c61388db0f972416e49b209c7b303de3d154c376834428e4da2ba0e5 not found: ID does not exist"
Sep 16 07:59:59.042147 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.040169    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/742901d8-bc38-432f-b3bd-8ad9e10a7ed7-plugins-config-volume\") pod \"742901d8-bc38-432f-b3bd-8ad9e10a7ed7\" (UID: \"742901d8-bc38-432f-b3bd-8ad9e10a7ed7\") "
Sep 16 07:59:59.042147 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.040239    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxvv7\" (UniqueName: \"kubernetes.io/projected/742901d8-bc38-432f-b3bd-8ad9e10a7ed7-kube-api-access-lxvv7\") pod \"742901d8-bc38-432f-b3bd-8ad9e10a7ed7\" (UID: \"742901d8-bc38-432f-b3bd-8ad9e10a7ed7\") "
Sep 16 07:59:59.042147 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.040291    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67dcac13-2152-4225-bfd1-782cd53568d1-plugins-config-volume\") pod \"67dcac13-2152-4225-bfd1-782cd53568d1\" (UID: \"67dcac13-2152-4225-bfd1-782cd53568d1\") "
Sep 16 07:59:59.042147 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.040365    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjmvg\" (UniqueName: \"kubernetes.io/projected/67dcac13-2152-4225-bfd1-782cd53568d1-kube-api-access-jjmvg\") pod \"67dcac13-2152-4225-bfd1-782cd53568d1\" (UID: \"67dcac13-2152-4225-bfd1-782cd53568d1\") "
Sep 16 07:59:59.042147 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.041798    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742901d8-bc38-432f-b3bd-8ad9e10a7ed7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "742901d8-bc38-432f-b3bd-8ad9e10a7ed7" (UID: "742901d8-bc38-432f-b3bd-8ad9e10a7ed7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:59:59.042147 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.041998    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67dcac13-2152-4225-bfd1-782cd53568d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67dcac13-2152-4225-bfd1-782cd53568d1" (UID: "67dcac13-2152-4225-bfd1-782cd53568d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 07:59:59.044585 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.044546    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67dcac13-2152-4225-bfd1-782cd53568d1-kube-api-access-jjmvg" (OuterVolumeSpecName: "kube-api-access-jjmvg") pod "67dcac13-2152-4225-bfd1-782cd53568d1" (UID: "67dcac13-2152-4225-bfd1-782cd53568d1"). InnerVolumeSpecName "kube-api-access-jjmvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:59:59.045614 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.045582    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742901d8-bc38-432f-b3bd-8ad9e10a7ed7-kube-api-access-lxvv7" (OuterVolumeSpecName: "kube-api-access-lxvv7") pod "742901d8-bc38-432f-b3bd-8ad9e10a7ed7" (UID: "742901d8-bc38-432f-b3bd-8ad9e10a7ed7"). InnerVolumeSpecName "kube-api-access-lxvv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 07:59:59.141802 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.141771    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/742901d8-bc38-432f-b3bd-8ad9e10a7ed7-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:59:59.141802 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.141797    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxvv7\" (UniqueName: \"kubernetes.io/projected/742901d8-bc38-432f-b3bd-8ad9e10a7ed7-kube-api-access-lxvv7\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:59:59.141802 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.141807    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67dcac13-2152-4225-bfd1-782cd53568d1-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:59:59.141802 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.141829    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjmvg\" (UniqueName: \"kubernetes.io/projected/67dcac13-2152-4225-bfd1-782cd53568d1-kube-api-access-jjmvg\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 07:59:59.288815 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.288781    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"]
Sep 16 07:59:59.292928 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.292894    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-987ea6-844c556647-rg5ns"]
Sep 16 07:59:59.305640 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.305607    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"]
Sep 16 07:59:59.309022 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.308992    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-987ea6-5dc548ff54-xn9z6"]
Sep 16 07:59:59.690892 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.690853    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e5dfc34-0ed6-4de6-846b-51992e01f54a" path="/var/lib/kubelet/pods/1e5dfc34-0ed6-4de6-846b-51992e01f54a/volumes"
Sep 16 07:59:59.691453 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.691432    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67dcac13-2152-4225-bfd1-782cd53568d1" path="/var/lib/kubelet/pods/67dcac13-2152-4225-bfd1-782cd53568d1/volumes"
Sep 16 07:59:59.691797 ip-10-0-139-143 kubenswrapper[2581]: I0916 07:59:59.691784    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="742901d8-bc38-432f-b3bd-8ad9e10a7ed7" path="/var/lib/kubelet/pods/742901d8-bc38-432f-b3bd-8ad9e10a7ed7/volumes"
Sep 16 08:00:00.149452 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.149362    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e9fc8c-29825760-5fn54"]
Sep 16 08:00:00.150091 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.149993    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e5dfc34-0ed6-4de6-846b-51992e01f54a" containerName="revoke-keys"
Sep 16 08:00:00.150091 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.150017    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e5dfc34-0ed6-4de6-846b-51992e01f54a" containerName="revoke-keys"
Sep 16 08:00:00.150091 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.150055    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742901d8-bc38-432f-b3bd-8ad9e10a7ed7" containerName="payload-pre-processing"
Sep 16 08:00:00.150091 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.150064    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="742901d8-bc38-432f-b3bd-8ad9e10a7ed7" containerName="payload-pre-processing"
Sep 16 08:00:00.150091 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.150080    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67dcac13-2152-4225-bfd1-782cd53568d1" containerName="payload-processing"
Sep 16 08:00:00.150091 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.150090    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="67dcac13-2152-4225-bfd1-782cd53568d1" containerName="payload-processing"
Sep 16 08:00:00.150476 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.150192    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="742901d8-bc38-432f-b3bd-8ad9e10a7ed7" containerName="payload-pre-processing"
Sep 16 08:00:00.150476 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.150208    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="67dcac13-2152-4225-bfd1-782cd53568d1" containerName="payload-processing"
Sep 16 08:00:00.150476 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.150218    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e5dfc34-0ed6-4de6-846b-51992e01f54a" containerName="revoke-keys"
Sep 16 08:00:00.153607 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.153579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e9fc8c-29825760-5fn54"
Sep 16 08:00:00.153862 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.153731    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a243c-29825760-jxtnf"]
Sep 16 08:00:00.156262 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.156243    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\""
Sep 16 08:00:00.157888 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.157867    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a243c-29825760-jxtnf"
Sep 16 08:00:00.168942 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.168645    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e9fc8c-29825760-5fn54"
Sep 16 08:00:00.169978 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.169876    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e9fc8c-29825760-5fn54"]
Sep 16 08:00:00.173284 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.173231    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a243c-29825760-jxtnf"]
Sep 16 08:00:00.181084 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.181051    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a243c-29825760-jxtnf"
Sep 16 08:00:00.181503 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.181478    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825760-h2trv"]
Sep 16 08:00:00.187183 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.187141    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825760-h2trv"
Sep 16 08:00:00.200783 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.200752    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825760-h2trv"]
Sep 16 08:00:00.230003 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.229975    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825760-h2trv"
Sep 16 08:00:00.477811 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.477786    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a243c-29825760-jxtnf"]
Sep 16 08:00:00.479830 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:00:00.479791    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ead664e_13c8_41d8_aeb8_f1e3b78f8157.slice/crio-85f80928af245c13cc7804363b3de7f400aaa166f028bb751862674c06b217fa WatchSource:0}: Error finding container 85f80928af245c13cc7804363b3de7f400aaa166f028bb751862674c06b217fa: Status 404 returned error can't find the container with id 85f80928af245c13cc7804363b3de7f400aaa166f028bb751862674c06b217fa
Sep 16 08:00:00.490162 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.490134    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e9fc8c-29825760-5fn54"]
Sep 16 08:00:00.495787 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:00:00.495752    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b48632_21db_4663_aae0_5e4cb613a35a.slice/crio-32e287c527055c6d6879be1f475d4933e8a6c08f4eee121b738007590a38ed79 WatchSource:0}: Error finding container 32e287c527055c6d6879be1f475d4933e8a6c08f4eee121b738007590a38ed79: Status 404 returned error can't find the container with id 32e287c527055c6d6879be1f475d4933e8a6c08f4eee121b738007590a38ed79
Sep 16 08:00:00.537939 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.537911    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825760-h2trv"]
Sep 16 08:00:00.539406 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:00:00.539378    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fbad26_b96a_4501_a82d_9b2d8e7e9c24.slice/crio-eb5cd8408d361885b0a5745976291934b3cd9739edc8a9d3647ade265d711bea WatchSource:0}: Error finding container eb5cd8408d361885b0a5745976291934b3cd9739edc8a9d3647ade265d711bea: Status 404 returned error can't find the container with id eb5cd8408d361885b0a5745976291934b3cd9739edc8a9d3647ade265d711bea
Sep 16 08:00:00.977134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.977100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a243c-29825760-jxtnf" event={"ID":"5ead664e-13c8-41d8-aeb8-f1e3b78f8157","Type":"ContainerStarted","Data":"db2357103f5464668bd7f4160d12b60eab4cfa328d7704843210d95217913514"}
Sep 16 08:00:00.977610 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.977161    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a243c-29825760-jxtnf" event={"ID":"5ead664e-13c8-41d8-aeb8-f1e3b78f8157","Type":"ContainerStarted","Data":"85f80928af245c13cc7804363b3de7f400aaa166f028bb751862674c06b217fa"}
Sep 16 08:00:00.978787 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.978756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825760-h2trv" event={"ID":"91fbad26-b96a-4501-a82d-9b2d8e7e9c24","Type":"ContainerStarted","Data":"303d94bd09aeebb639e99fab7c4c30cb21edfc4db1f8c752b2cf9f2b35f2a13d"}
Sep 16 08:00:00.978929 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.978806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825760-h2trv" event={"ID":"91fbad26-b96a-4501-a82d-9b2d8e7e9c24","Type":"ContainerStarted","Data":"eb5cd8408d361885b0a5745976291934b3cd9739edc8a9d3647ade265d711bea"}
Sep 16 08:00:00.980266 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.980240    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e9fc8c-29825760-5fn54" event={"ID":"b3b48632-21db-4663-aae0-5e4cb613a35a","Type":"ContainerStarted","Data":"0c2a9885e955bac95d40a4d057db14d5a1f770549a398fc66acd102c013cfcda"}
Sep 16 08:00:00.980381 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.980280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e9fc8c-29825760-5fn54" event={"ID":"b3b48632-21db-4663-aae0-5e4cb613a35a","Type":"ContainerStarted","Data":"32e287c527055c6d6879be1f475d4933e8a6c08f4eee121b738007590a38ed79"}
Sep 16 08:00:00.993302 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:00.993200    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a243c-29825760-jxtnf" podStartSLOduration=0.993184187 podStartE2EDuration="993.184187ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:00:00.991632332 +0000 UTC m=+1561.918244479" watchObservedRunningTime="2026-09-16 08:00:00.993184187 +0000 UTC m=+1561.919796340"
Sep 16 08:00:01.007534 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:01.007473    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825760-h2trv" podStartSLOduration=1.007452533 podStartE2EDuration="1.007452533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:00:01.007349144 +0000 UTC m=+1561.933961298" watchObservedRunningTime="2026-09-16 08:00:01.007452533 +0000 UTC m=+1561.934064687"
Sep 16 08:00:01.022467 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:01.022409    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e9fc8c-29825760-5fn54" podStartSLOduration=1.022390341 podStartE2EDuration="1.022390341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:00:01.020577692 +0000 UTC m=+1561.947189847" watchObservedRunningTime="2026-09-16 08:00:01.022390341 +0000 UTC m=+1561.949002493"
Sep 16 08:00:01.987485 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:01.987450    2581 generic.go:356] "Generic (PLEG): container finished" podID="5ead664e-13c8-41d8-aeb8-f1e3b78f8157" containerID="db2357103f5464668bd7f4160d12b60eab4cfa328d7704843210d95217913514" exitCode=0
Sep 16 08:00:01.987913 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:01.987544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a243c-29825760-jxtnf" event={"ID":"5ead664e-13c8-41d8-aeb8-f1e3b78f8157","Type":"ContainerDied","Data":"db2357103f5464668bd7f4160d12b60eab4cfa328d7704843210d95217913514"}
Sep 16 08:00:01.989035 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:01.989010    2581 generic.go:356] "Generic (PLEG): container finished" podID="91fbad26-b96a-4501-a82d-9b2d8e7e9c24" containerID="303d94bd09aeebb639e99fab7c4c30cb21edfc4db1f8c752b2cf9f2b35f2a13d" exitCode=0
Sep 16 08:00:01.989145 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:01.989098    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825760-h2trv" event={"ID":"91fbad26-b96a-4501-a82d-9b2d8e7e9c24","Type":"ContainerDied","Data":"303d94bd09aeebb639e99fab7c4c30cb21edfc4db1f8c752b2cf9f2b35f2a13d"}
Sep 16 08:00:01.990521 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:01.990498    2581 generic.go:356] "Generic (PLEG): container finished" podID="b3b48632-21db-4663-aae0-5e4cb613a35a" containerID="0c2a9885e955bac95d40a4d057db14d5a1f770549a398fc66acd102c013cfcda" exitCode=0
Sep 16 08:00:01.990621 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:01.990574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e9fc8c-29825760-5fn54" event={"ID":"b3b48632-21db-4663-aae0-5e4cb613a35a","Type":"ContainerDied","Data":"0c2a9885e955bac95d40a4d057db14d5a1f770549a398fc66acd102c013cfcda"}
Sep 16 08:00:03.148559 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:03.148533    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e9fc8c-29825760-5fn54"
Sep 16 08:00:03.226526 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:03.226497    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a243c-29825760-jxtnf"
Sep 16 08:00:03.229757 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:03.229731    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825760-h2trv"
Sep 16 08:00:04.003720 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:04.003635    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a243c-29825760-jxtnf"
Sep 16 08:00:04.003720 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:04.003641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a243c-29825760-jxtnf" event={"ID":"5ead664e-13c8-41d8-aeb8-f1e3b78f8157","Type":"ContainerDied","Data":"85f80928af245c13cc7804363b3de7f400aaa166f028bb751862674c06b217fa"}
Sep 16 08:00:04.003966 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:04.003731    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85f80928af245c13cc7804363b3de7f400aaa166f028bb751862674c06b217fa"
Sep 16 08:00:04.005113 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:04.005093    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825760-h2trv"
Sep 16 08:00:04.005238 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:04.005119    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825760-h2trv" event={"ID":"91fbad26-b96a-4501-a82d-9b2d8e7e9c24","Type":"ContainerDied","Data":"eb5cd8408d361885b0a5745976291934b3cd9739edc8a9d3647ade265d711bea"}
Sep 16 08:00:04.005238 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:04.005159    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb5cd8408d361885b0a5745976291934b3cd9739edc8a9d3647ade265d711bea"
Sep 16 08:00:04.006580 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:04.006561    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e9fc8c-29825760-5fn54" event={"ID":"b3b48632-21db-4663-aae0-5e4cb613a35a","Type":"ContainerDied","Data":"32e287c527055c6d6879be1f475d4933e8a6c08f4eee121b738007590a38ed79"}
Sep 16 08:00:04.006651 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:04.006596    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32e287c527055c6d6879be1f475d4933e8a6c08f4eee121b738007590a38ed79"
Sep 16 08:00:04.006651 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:04.006614    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e9fc8c-29825760-5fn54"
Sep 16 08:00:04.441681 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:04.441648    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"]
Sep 16 08:00:04.442486 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:04.442456    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c" podUID="a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" containerName="istio-proxy" containerID="cri-o://d7ceffd5d8b9890e3b08d56b575cbdebc3887f29144a32b18248a1914a550957" gracePeriod=30
Sep 16 08:00:06.107262 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:06.107229    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"]
Sep 16 08:00:06.107737 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:06.107462    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk" podUID="ca0fdf27-753f-4005-847b-68c869d325ae" containerName="istio-proxy" containerID="cri-o://2389c24d01450295bc671e19f63022117a582bbd03f3d8d54c2dcfbf145656b1" gracePeriod=30
Sep 16 08:00:06.920257 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:06.920216    2581 patch_prober.go:28] interesting pod/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 08:00:06.920455 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:06.920311    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk" podUID="ca0fdf27-753f-4005-847b-68c869d325ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 08:00:09.717839 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.717767    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 08:00:09.867033 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867004    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-data\") pod \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") "
Sep 16 08:00:09.867207 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867048    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-workload-certs\") pod \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") "
Sep 16 08:00:09.867207 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867085    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-token\") pod \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") "
Sep 16 08:00:09.867207 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867102    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-envoy\") pod \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") "
Sep 16 08:00:09.867207 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867142    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vn99\" (UniqueName: \"kubernetes.io/projected/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-kube-api-access-4vn99\") pod \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") "
Sep 16 08:00:09.867207 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867181    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-podinfo\") pod \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") "
Sep 16 08:00:09.867471 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867229    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-workload-socket\") pod \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") "
Sep 16 08:00:09.867471 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867340    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-credential-socket\") pod \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") "
Sep 16 08:00:09.867471 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867376    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istiod-ca-cert\") pod \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\" (UID: \"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b\") "
Sep 16 08:00:09.867646 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867570    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" (UID: "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:09.867646 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867592    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-data" (OuterVolumeSpecName: "istio-data") pod "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" (UID: "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:09.867752 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867644    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" (UID: "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:09.867807 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867792    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" (UID: "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:09.867932 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867914    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:09.867991 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867953    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:09.867991 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.867967    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:09.868279 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.868006    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" (UID: "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:00:09.869675 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.869641    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-kube-api-access-4vn99" (OuterVolumeSpecName: "kube-api-access-4vn99") pod "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" (UID: "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b"). InnerVolumeSpecName "kube-api-access-4vn99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:00:09.869675 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.869648    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-token" (OuterVolumeSpecName: "istio-token") pod "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" (UID: "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:00:09.869864 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.869678    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" (UID: "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:09.869930 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.869900    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" (UID: "a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 08:00:09.975847 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.970396    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:09.975847 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.970428    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:09.975847 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.970451    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:09.975847 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.970464    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:09.975847 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.970477    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vn99\" (UniqueName: \"kubernetes.io/projected/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-kube-api-access-4vn99\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:09.975847 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:09.970492    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:10.042014 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:10.041985    2581 generic.go:356] "Generic (PLEG): container finished" podID="a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" containerID="d7ceffd5d8b9890e3b08d56b575cbdebc3887f29144a32b18248a1914a550957" exitCode=0
Sep 16 08:00:10.042153 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:10.042043    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"
Sep 16 08:00:10.042153 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:10.042044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c" event={"ID":"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b","Type":"ContainerDied","Data":"d7ceffd5d8b9890e3b08d56b575cbdebc3887f29144a32b18248a1914a550957"}
Sep 16 08:00:10.042266 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:10.042197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c" event={"ID":"a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b","Type":"ContainerDied","Data":"bbf1bc672cee355529a4f2f6483f4e4dbd3c7dafe5c34da02532b5beb5b1f67a"}
Sep 16 08:00:10.042266 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:10.042225    2581 scope.go:117] "RemoveContainer" containerID="d7ceffd5d8b9890e3b08d56b575cbdebc3887f29144a32b18248a1914a550957"
Sep 16 08:00:10.052387 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:10.052370    2581 scope.go:117] "RemoveContainer" containerID="33f02d1e40305976363c2776efc7b306cc12123359f6debb47a5b1cbc10d7bf1"
Sep 16 08:00:10.060299 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:10.060285    2581 scope.go:117] "RemoveContainer" containerID="d7ceffd5d8b9890e3b08d56b575cbdebc3887f29144a32b18248a1914a550957"
Sep 16 08:00:10.060534 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:00:10.060517    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ceffd5d8b9890e3b08d56b575cbdebc3887f29144a32b18248a1914a550957\": container with ID starting with d7ceffd5d8b9890e3b08d56b575cbdebc3887f29144a32b18248a1914a550957 not found: ID does not exist" containerID="d7ceffd5d8b9890e3b08d56b575cbdebc3887f29144a32b18248a1914a550957"
Sep 16 08:00:10.060578 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:10.060541    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ceffd5d8b9890e3b08d56b575cbdebc3887f29144a32b18248a1914a550957"} err="failed to get container status \"d7ceffd5d8b9890e3b08d56b575cbdebc3887f29144a32b18248a1914a550957\": rpc error: code = NotFound desc = could not find container \"d7ceffd5d8b9890e3b08d56b575cbdebc3887f29144a32b18248a1914a550957\": container with ID starting with d7ceffd5d8b9890e3b08d56b575cbdebc3887f29144a32b18248a1914a550957 not found: ID does not exist"
Sep 16 08:00:10.060578 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:10.060558    2581 scope.go:117] "RemoveContainer" containerID="33f02d1e40305976363c2776efc7b306cc12123359f6debb47a5b1cbc10d7bf1"
Sep 16 08:00:10.060766 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:00:10.060750    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f02d1e40305976363c2776efc7b306cc12123359f6debb47a5b1cbc10d7bf1\": container with ID starting with 33f02d1e40305976363c2776efc7b306cc12123359f6debb47a5b1cbc10d7bf1 not found: ID does not exist" containerID="33f02d1e40305976363c2776efc7b306cc12123359f6debb47a5b1cbc10d7bf1"
Sep 16 08:00:10.060808 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:10.060771    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f02d1e40305976363c2776efc7b306cc12123359f6debb47a5b1cbc10d7bf1"} err="failed to get container status \"33f02d1e40305976363c2776efc7b306cc12123359f6debb47a5b1cbc10d7bf1\": rpc error: code = NotFound desc = could not find container \"33f02d1e40305976363c2776efc7b306cc12123359f6debb47a5b1cbc10d7bf1\": container with ID starting with 33f02d1e40305976363c2776efc7b306cc12123359f6debb47a5b1cbc10d7bf1 not found: ID does not exist"
Sep 16 08:00:10.064810 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:10.064791    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"]
Sep 16 08:00:10.069210 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:10.069189    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c9020d-openshift-default-788fb6bbbd-h5s7c"]
Sep 16 08:00:11.366922 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.366865    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 08:00:11.484491 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.484462    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0fdf27-753f-4005-847b-68c869d325ae-istio-token\") pod \"ca0fdf27-753f-4005-847b-68c869d325ae\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") "
Sep 16 08:00:11.484639 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.484501    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-istio-data\") pod \"ca0fdf27-753f-4005-847b-68c869d325ae\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") "
Sep 16 08:00:11.484639 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.484551    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2dcs\" (UniqueName: \"kubernetes.io/projected/ca0fdf27-753f-4005-847b-68c869d325ae-kube-api-access-t2dcs\") pod \"ca0fdf27-753f-4005-847b-68c869d325ae\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") "
Sep 16 08:00:11.484639 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.484586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-workload-certs\") pod \"ca0fdf27-753f-4005-847b-68c869d325ae\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") "
Sep 16 08:00:11.484639 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.484604    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0fdf27-753f-4005-847b-68c869d325ae-istio-podinfo\") pod \"ca0fdf27-753f-4005-847b-68c869d325ae\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") "
Sep 16 08:00:11.484639 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.484625    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0fdf27-753f-4005-847b-68c869d325ae-istiod-ca-cert\") pod \"ca0fdf27-753f-4005-847b-68c869d325ae\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") "
Sep 16 08:00:11.484931 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.484694    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-istio-envoy\") pod \"ca0fdf27-753f-4005-847b-68c869d325ae\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") "
Sep 16 08:00:11.484931 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.484720    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-workload-socket\") pod \"ca0fdf27-753f-4005-847b-68c869d325ae\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") "
Sep 16 08:00:11.484931 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.484753    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-credential-socket\") pod \"ca0fdf27-753f-4005-847b-68c869d325ae\" (UID: \"ca0fdf27-753f-4005-847b-68c869d325ae\") "
Sep 16 08:00:11.484931 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.484864    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca0fdf27-753f-4005-847b-68c869d325ae" (UID: "ca0fdf27-753f-4005-847b-68c869d325ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:11.484931 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.484925    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca0fdf27-753f-4005-847b-68c869d325ae" (UID: "ca0fdf27-753f-4005-847b-68c869d325ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:11.485182 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.485082    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca0fdf27-753f-4005-847b-68c869d325ae" (UID: "ca0fdf27-753f-4005-847b-68c869d325ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:11.485182 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.485101    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca0fdf27-753f-4005-847b-68c869d325ae" (UID: "ca0fdf27-753f-4005-847b-68c869d325ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:11.485300 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.485227    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:11.485300 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.485247    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:11.485300 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.485260    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:11.485300 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.485273    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:11.485474 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.485355    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca0fdf27-753f-4005-847b-68c869d325ae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca0fdf27-753f-4005-847b-68c869d325ae" (UID: "ca0fdf27-753f-4005-847b-68c869d325ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:00:11.486990 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.486967    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0fdf27-753f-4005-847b-68c869d325ae-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca0fdf27-753f-4005-847b-68c869d325ae" (UID: "ca0fdf27-753f-4005-847b-68c869d325ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:00:11.487102 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.487085    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca0fdf27-753f-4005-847b-68c869d325ae" (UID: "ca0fdf27-753f-4005-847b-68c869d325ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:11.487160 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.487107    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0fdf27-753f-4005-847b-68c869d325ae-kube-api-access-t2dcs" (OuterVolumeSpecName: "kube-api-access-t2dcs") pod "ca0fdf27-753f-4005-847b-68c869d325ae" (UID: "ca0fdf27-753f-4005-847b-68c869d325ae"). InnerVolumeSpecName "kube-api-access-t2dcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:00:11.487203 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.487176    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca0fdf27-753f-4005-847b-68c869d325ae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca0fdf27-753f-4005-847b-68c869d325ae" (UID: "ca0fdf27-753f-4005-847b-68c869d325ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 08:00:11.586101 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.586068    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0fdf27-753f-4005-847b-68c869d325ae-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:11.586101 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.586095    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0fdf27-753f-4005-847b-68c869d325ae-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:11.586101 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.586105    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2dcs\" (UniqueName: \"kubernetes.io/projected/ca0fdf27-753f-4005-847b-68c869d325ae-kube-api-access-t2dcs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:11.586343 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.586116    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0fdf27-753f-4005-847b-68c869d325ae-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:11.586343 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.586126    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0fdf27-753f-4005-847b-68c869d325ae-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:11.684529 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:11.684495    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" path="/var/lib/kubelet/pods/a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b/volumes"
Sep 16 08:00:12.055845 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:12.055733    2581 generic.go:356] "Generic (PLEG): container finished" podID="ca0fdf27-753f-4005-847b-68c869d325ae" containerID="2389c24d01450295bc671e19f63022117a582bbd03f3d8d54c2dcfbf145656b1" exitCode=0
Sep 16 08:00:12.055845 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:12.055804    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"
Sep 16 08:00:12.056067 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:12.055842    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk" event={"ID":"ca0fdf27-753f-4005-847b-68c869d325ae","Type":"ContainerDied","Data":"2389c24d01450295bc671e19f63022117a582bbd03f3d8d54c2dcfbf145656b1"}
Sep 16 08:00:12.056067 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:12.055947    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk" event={"ID":"ca0fdf27-753f-4005-847b-68c869d325ae","Type":"ContainerDied","Data":"2841a3bdb20427ef6cdf70558343ac31ddca01a6da8147eccdce19ba966ebdf4"}
Sep 16 08:00:12.056067 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:12.055978    2581 scope.go:117] "RemoveContainer" containerID="2389c24d01450295bc671e19f63022117a582bbd03f3d8d54c2dcfbf145656b1"
Sep 16 08:00:12.066122 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:12.066094    2581 scope.go:117] "RemoveContainer" containerID="26602db4ca79043b7818384b565149ef4474f358e9dae9f2a287b62c0adda2ef"
Sep 16 08:00:12.074273 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:12.074251    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"]
Sep 16 08:00:12.079935 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:12.079914    2581 scope.go:117] "RemoveContainer" containerID="2389c24d01450295bc671e19f63022117a582bbd03f3d8d54c2dcfbf145656b1"
Sep 16 08:00:12.080265 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:00:12.080246    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2389c24d01450295bc671e19f63022117a582bbd03f3d8d54c2dcfbf145656b1\": container with ID starting with 2389c24d01450295bc671e19f63022117a582bbd03f3d8d54c2dcfbf145656b1 not found: ID does not exist" containerID="2389c24d01450295bc671e19f63022117a582bbd03f3d8d54c2dcfbf145656b1"
Sep 16 08:00:12.080327 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:12.080274    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2389c24d01450295bc671e19f63022117a582bbd03f3d8d54c2dcfbf145656b1"} err="failed to get container status \"2389c24d01450295bc671e19f63022117a582bbd03f3d8d54c2dcfbf145656b1\": rpc error: code = NotFound desc = could not find container \"2389c24d01450295bc671e19f63022117a582bbd03f3d8d54c2dcfbf145656b1\": container with ID starting with 2389c24d01450295bc671e19f63022117a582bbd03f3d8d54c2dcfbf145656b1 not found: ID does not exist"
Sep 16 08:00:12.080327 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:12.080292    2581 scope.go:117] "RemoveContainer" containerID="26602db4ca79043b7818384b565149ef4474f358e9dae9f2a287b62c0adda2ef"
Sep 16 08:00:12.080546 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:00:12.080531    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26602db4ca79043b7818384b565149ef4474f358e9dae9f2a287b62c0adda2ef\": container with ID starting with 26602db4ca79043b7818384b565149ef4474f358e9dae9f2a287b62c0adda2ef not found: ID does not exist" containerID="26602db4ca79043b7818384b565149ef4474f358e9dae9f2a287b62c0adda2ef"
Sep 16 08:00:12.080632 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:12.080550    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26602db4ca79043b7818384b565149ef4474f358e9dae9f2a287b62c0adda2ef"} err="failed to get container status \"26602db4ca79043b7818384b565149ef4474f358e9dae9f2a287b62c0adda2ef\": rpc error: code = NotFound desc = could not find container \"26602db4ca79043b7818384b565149ef4474f358e9dae9f2a287b62c0adda2ef\": container with ID starting with 26602db4ca79043b7818384b565149ef4474f358e9dae9f2a287b62c0adda2ef not found: ID does not exist"
Sep 16 08:00:12.081774 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:12.081752    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-987ea6-openshift-default-7886b989dd-zr5kk"]
Sep 16 08:00:13.685010 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:13.684976    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca0fdf27-753f-4005-847b-68c869d325ae" path="/var/lib/kubelet/pods/ca0fdf27-753f-4005-847b-68c869d325ae/volumes"
Sep 16 08:00:28.647333 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:28.647298    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"]
Sep 16 08:00:28.647770 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:28.647605    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8" podUID="d391f1c7-023e-485b-8709-7f0e4336eafc" containerName="istio-proxy" containerID="cri-o://7e9b5d1021a71620431a32070de00a28a3104fefe8b3620f95332de307769e45" gracePeriod=30
Sep 16 08:00:30.349704 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.349663    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2"]
Sep 16 08:00:30.350298 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350276    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" containerName="istio-proxy"
Sep 16 08:00:30.350394 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350300    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" containerName="istio-proxy"
Sep 16 08:00:30.350394 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350319    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ead664e-13c8-41d8-aeb8-f1e3b78f8157" containerName="cleanup"
Sep 16 08:00:30.350394 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350327    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ead664e-13c8-41d8-aeb8-f1e3b78f8157" containerName="cleanup"
Sep 16 08:00:30.350394 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350339    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3b48632-21db-4663-aae0-5e4cb613a35a" containerName="cleanup"
Sep 16 08:00:30.350394 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350347    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3b48632-21db-4663-aae0-5e4cb613a35a" containerName="cleanup"
Sep 16 08:00:30.350394 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350369    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" containerName="istio-proxy"
Sep 16 08:00:30.350394 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350376    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" containerName="istio-proxy"
Sep 16 08:00:30.350394 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350388    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca0fdf27-753f-4005-847b-68c869d325ae" containerName="istio-proxy"
Sep 16 08:00:30.350394 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350396    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca0fdf27-753f-4005-847b-68c869d325ae" containerName="istio-proxy"
Sep 16 08:00:30.350862 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350416    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca0fdf27-753f-4005-847b-68c869d325ae" containerName="istio-proxy"
Sep 16 08:00:30.350862 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350425    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca0fdf27-753f-4005-847b-68c869d325ae" containerName="istio-proxy"
Sep 16 08:00:30.350862 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350435    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91fbad26-b96a-4501-a82d-9b2d8e7e9c24" containerName="cleanup"
Sep 16 08:00:30.350862 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350443    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="91fbad26-b96a-4501-a82d-9b2d8e7e9c24" containerName="cleanup"
Sep 16 08:00:30.350862 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350547    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca0fdf27-753f-4005-847b-68c869d325ae" containerName="istio-proxy"
Sep 16 08:00:30.350862 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350558    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="91fbad26-b96a-4501-a82d-9b2d8e7e9c24" containerName="cleanup"
Sep 16 08:00:30.350862 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350570    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" containerName="istio-proxy"
Sep 16 08:00:30.350862 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350581    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60a8a8f-b2a4-40e7-88fb-ffee17f9a01b" containerName="istio-proxy"
Sep 16 08:00:30.350862 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350589    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca0fdf27-753f-4005-847b-68c869d325ae" containerName="istio-proxy"
Sep 16 08:00:30.350862 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350600    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ead664e-13c8-41d8-aeb8-f1e3b78f8157" containerName="cleanup"
Sep 16 08:00:30.350862 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.350610    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3b48632-21db-4663-aae0-5e4cb613a35a" containerName="cleanup"
Sep 16 08:00:30.355603 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.355582    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2"
Sep 16 08:00:30.358173 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.358149    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\""
Sep 16 08:00:30.358488 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.358226    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 08:00:30.360400 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.360376    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2"]
Sep 16 08:00:30.381329 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.381306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a6933b-e889-4ae6-83d8-d512efc75536-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2\" (UID: \"f2a6933b-e889-4ae6-83d8-d512efc75536\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2"
Sep 16 08:00:30.482481 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.482453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a6933b-e889-4ae6-83d8-d512efc75536-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2\" (UID: \"f2a6933b-e889-4ae6-83d8-d512efc75536\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2"
Sep 16 08:00:30.483058 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.483042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a6933b-e889-4ae6-83d8-d512efc75536-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2\" (UID: \"f2a6933b-e889-4ae6-83d8-d512efc75536\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2"
Sep 16 08:00:30.668071 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.668045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2"
Sep 16 08:00:30.938202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:30.938174    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2"]
Sep 16 08:00:30.939539 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:00:30.939514    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a6933b_e889_4ae6_83d8_d512efc75536.slice/crio-75cb755557f67ea1a05f95846969628e90a0eb29f46ebcd8a87dfebb4a299fdc WatchSource:0}: Error finding container 75cb755557f67ea1a05f95846969628e90a0eb29f46ebcd8a87dfebb4a299fdc: Status 404 returned error can't find the container with id 75cb755557f67ea1a05f95846969628e90a0eb29f46ebcd8a87dfebb4a299fdc
Sep 16 08:00:31.179137 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:31.179104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2" event={"ID":"f2a6933b-e889-4ae6-83d8-d512efc75536","Type":"ContainerStarted","Data":"75cb755557f67ea1a05f95846969628e90a0eb29f46ebcd8a87dfebb4a299fdc"}
Sep 16 08:00:32.186361 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:32.186325    2581 generic.go:356] "Generic (PLEG): container finished" podID="f2a6933b-e889-4ae6-83d8-d512efc75536" containerID="24e15325d5ae77cc62478889348c5412b8a9219344062aa2cf3cf3052569bc55" exitCode=0
Sep 16 08:00:32.186766 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:32.186410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2" event={"ID":"f2a6933b-e889-4ae6-83d8-d512efc75536","Type":"ContainerDied","Data":"24e15325d5ae77cc62478889348c5412b8a9219344062aa2cf3cf3052569bc55"}
Sep 16 08:00:33.331641 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:33.331612    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2"
Sep 16 08:00:33.410767 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:33.410653    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a6933b-e889-4ae6-83d8-d512efc75536-maas-api-service-ca\") pod \"f2a6933b-e889-4ae6-83d8-d512efc75536\" (UID: \"f2a6933b-e889-4ae6-83d8-d512efc75536\") "
Sep 16 08:00:33.411860 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:33.411564    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a6933b-e889-4ae6-83d8-d512efc75536-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f2a6933b-e889-4ae6-83d8-d512efc75536" (UID: "f2a6933b-e889-4ae6-83d8-d512efc75536"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:00:33.512056 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:33.511981    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a6933b-e889-4ae6-83d8-d512efc75536-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:33.901929 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:33.901907    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 08:00:34.016799 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.016726    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-credential-socket\") pod \"d391f1c7-023e-485b-8709-7f0e4336eafc\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") "
Sep 16 08:00:34.016799 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.016756    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-workload-socket\") pod \"d391f1c7-023e-485b-8709-7f0e4336eafc\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") "
Sep 16 08:00:34.016799 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.016784    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-podinfo\") pod \"d391f1c7-023e-485b-8709-7f0e4336eafc\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") "
Sep 16 08:00:34.017100 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.016865    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-envoy\") pod \"d391f1c7-023e-485b-8709-7f0e4336eafc\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") "
Sep 16 08:00:34.017100 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.016884    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-token\") pod \"d391f1c7-023e-485b-8709-7f0e4336eafc\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") "
Sep 16 08:00:34.017100 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.016912    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5qz6\" (UniqueName: \"kubernetes.io/projected/d391f1c7-023e-485b-8709-7f0e4336eafc-kube-api-access-g5qz6\") pod \"d391f1c7-023e-485b-8709-7f0e4336eafc\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") "
Sep 16 08:00:34.017100 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.016992    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-workload-certs\") pod \"d391f1c7-023e-485b-8709-7f0e4336eafc\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") "
Sep 16 08:00:34.017100 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.017013    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d391f1c7-023e-485b-8709-7f0e4336eafc-istiod-ca-cert\") pod \"d391f1c7-023e-485b-8709-7f0e4336eafc\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") "
Sep 16 08:00:34.017100 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.017053    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-data\") pod \"d391f1c7-023e-485b-8709-7f0e4336eafc\" (UID: \"d391f1c7-023e-485b-8709-7f0e4336eafc\") "
Sep 16 08:00:34.017100 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.016984    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d391f1c7-023e-485b-8709-7f0e4336eafc" (UID: "d391f1c7-023e-485b-8709-7f0e4336eafc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:34.017100 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.017085    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d391f1c7-023e-485b-8709-7f0e4336eafc" (UID: "d391f1c7-023e-485b-8709-7f0e4336eafc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:34.017514 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.017304    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d391f1c7-023e-485b-8709-7f0e4336eafc" (UID: "d391f1c7-023e-485b-8709-7f0e4336eafc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:34.017514 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.017325    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-data" (OuterVolumeSpecName: "istio-data") pod "d391f1c7-023e-485b-8709-7f0e4336eafc" (UID: "d391f1c7-023e-485b-8709-7f0e4336eafc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:34.017514 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.017460    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:34.017514 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.017481    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:34.017514 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.017494    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:34.017788 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.017528    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:34.017788 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.017488    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d391f1c7-023e-485b-8709-7f0e4336eafc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d391f1c7-023e-485b-8709-7f0e4336eafc" (UID: "d391f1c7-023e-485b-8709-7f0e4336eafc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:00:34.019211 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.019182    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d391f1c7-023e-485b-8709-7f0e4336eafc-kube-api-access-g5qz6" (OuterVolumeSpecName: "kube-api-access-g5qz6") pod "d391f1c7-023e-485b-8709-7f0e4336eafc" (UID: "d391f1c7-023e-485b-8709-7f0e4336eafc"). InnerVolumeSpecName "kube-api-access-g5qz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:00:34.019321 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.019226    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-token" (OuterVolumeSpecName: "istio-token") pod "d391f1c7-023e-485b-8709-7f0e4336eafc" (UID: "d391f1c7-023e-485b-8709-7f0e4336eafc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:00:34.019321 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.019282    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d391f1c7-023e-485b-8709-7f0e4336eafc" (UID: "d391f1c7-023e-485b-8709-7f0e4336eafc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 08:00:34.019321 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.019287    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d391f1c7-023e-485b-8709-7f0e4336eafc" (UID: "d391f1c7-023e-485b-8709-7f0e4336eafc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:34.118292 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.118256    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:34.118292 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.118285    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:34.118292 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.118293    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d391f1c7-023e-485b-8709-7f0e4336eafc-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:34.118292 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.118302    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5qz6\" (UniqueName: \"kubernetes.io/projected/d391f1c7-023e-485b-8709-7f0e4336eafc-kube-api-access-g5qz6\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:34.118558 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.118312    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d391f1c7-023e-485b-8709-7f0e4336eafc-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:34.199074 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.199038    2581 generic.go:356] "Generic (PLEG): container finished" podID="d391f1c7-023e-485b-8709-7f0e4336eafc" containerID="7e9b5d1021a71620431a32070de00a28a3104fefe8b3620f95332de307769e45" exitCode=0
Sep 16 08:00:34.199225 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.199080    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"
Sep 16 08:00:34.199225 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.199121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8" event={"ID":"d391f1c7-023e-485b-8709-7f0e4336eafc","Type":"ContainerDied","Data":"7e9b5d1021a71620431a32070de00a28a3104fefe8b3620f95332de307769e45"}
Sep 16 08:00:34.199225 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.199173    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8" event={"ID":"d391f1c7-023e-485b-8709-7f0e4336eafc","Type":"ContainerDied","Data":"bf7c19e452d391b9f15316affa2f97adfca92a8991eec2a4279b72b6eba6309d"}
Sep 16 08:00:34.199225 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.199196    2581 scope.go:117] "RemoveContainer" containerID="7e9b5d1021a71620431a32070de00a28a3104fefe8b3620f95332de307769e45"
Sep 16 08:00:34.200679 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.200657    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2" event={"ID":"f2a6933b-e889-4ae6-83d8-d512efc75536","Type":"ContainerDied","Data":"75cb755557f67ea1a05f95846969628e90a0eb29f46ebcd8a87dfebb4a299fdc"}
Sep 16 08:00:34.200746 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.200705    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75cb755557f67ea1a05f95846969628e90a0eb29f46ebcd8a87dfebb4a299fdc"
Sep 16 08:00:34.200746 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.200714    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2"
Sep 16 08:00:34.208855 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.208840    2581 scope.go:117] "RemoveContainer" containerID="848517fec7ab75f1bc63f6b21de3f5569a0b11e7fa88fc3a3ef785f988a8f554"
Sep 16 08:00:34.216904 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.216863    2581 scope.go:117] "RemoveContainer" containerID="7e9b5d1021a71620431a32070de00a28a3104fefe8b3620f95332de307769e45"
Sep 16 08:00:34.217199 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:00:34.217179    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e9b5d1021a71620431a32070de00a28a3104fefe8b3620f95332de307769e45\": container with ID starting with 7e9b5d1021a71620431a32070de00a28a3104fefe8b3620f95332de307769e45 not found: ID does not exist" containerID="7e9b5d1021a71620431a32070de00a28a3104fefe8b3620f95332de307769e45"
Sep 16 08:00:34.217297 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.217205    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e9b5d1021a71620431a32070de00a28a3104fefe8b3620f95332de307769e45"} err="failed to get container status \"7e9b5d1021a71620431a32070de00a28a3104fefe8b3620f95332de307769e45\": rpc error: code = NotFound desc = could not find container \"7e9b5d1021a71620431a32070de00a28a3104fefe8b3620f95332de307769e45\": container with ID starting with 7e9b5d1021a71620431a32070de00a28a3104fefe8b3620f95332de307769e45 not found: ID does not exist"
Sep 16 08:00:34.217297 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.217254    2581 scope.go:117] "RemoveContainer" containerID="848517fec7ab75f1bc63f6b21de3f5569a0b11e7fa88fc3a3ef785f988a8f554"
Sep 16 08:00:34.217552 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:00:34.217534    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848517fec7ab75f1bc63f6b21de3f5569a0b11e7fa88fc3a3ef785f988a8f554\": container with ID starting with 848517fec7ab75f1bc63f6b21de3f5569a0b11e7fa88fc3a3ef785f988a8f554 not found: ID does not exist" containerID="848517fec7ab75f1bc63f6b21de3f5569a0b11e7fa88fc3a3ef785f988a8f554"
Sep 16 08:00:34.217628 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.217557    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848517fec7ab75f1bc63f6b21de3f5569a0b11e7fa88fc3a3ef785f988a8f554"} err="failed to get container status \"848517fec7ab75f1bc63f6b21de3f5569a0b11e7fa88fc3a3ef785f988a8f554\": rpc error: code = NotFound desc = could not find container \"848517fec7ab75f1bc63f6b21de3f5569a0b11e7fa88fc3a3ef785f988a8f554\": container with ID starting with 848517fec7ab75f1bc63f6b21de3f5569a0b11e7fa88fc3a3ef785f988a8f554 not found: ID does not exist"
Sep 16 08:00:34.223857 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.223835    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"]
Sep 16 08:00:34.229117 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:34.229095    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3a5b4c-openshift-default-845665fd4b-sjvx8"]
Sep 16 08:00:35.218870 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:35.218838    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"]
Sep 16 08:00:35.684062 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:35.684026    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d391f1c7-023e-485b-8709-7f0e4336eafc" path="/var/lib/kubelet/pods/d391f1c7-023e-485b-8709-7f0e4336eafc/volumes"
Sep 16 08:00:39.684500 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:39.684469    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a243c-29825760-jxtnf"]
Sep 16 08:00:39.684899 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:39.684851    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4a243c-29825760-jxtnf"]
Sep 16 08:00:39.721375 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:39.721339    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"]
Sep 16 08:00:39.721651 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:39.721624    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd" podUID="79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e" containerName="payload-pre-processing" containerID="cri-o://1ed92a2896c8e4f1294cbe677f3bfe54fba4258981023ecab5073f011f043c18" gracePeriod=30
Sep 16 08:00:40.015635 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.015605    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"
Sep 16 08:00:40.076738 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.076698    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5k6r5\" (UniqueName: \"kubernetes.io/projected/79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e-kube-api-access-5k6r5\") pod \"79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e\" (UID: \"79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e\") "
Sep 16 08:00:40.076930 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.076909    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e-plugins-config-volume\") pod \"79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e\" (UID: \"79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e\") "
Sep 16 08:00:40.077267 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.077243    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e" (UID: "79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:00:40.078963 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.078939    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e-kube-api-access-5k6r5" (OuterVolumeSpecName: "kube-api-access-5k6r5") pod "79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e" (UID: "79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e"). InnerVolumeSpecName "kube-api-access-5k6r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:00:40.178599 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.178556    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:40.178599 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.178597    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k6r5\" (UniqueName: \"kubernetes.io/projected/79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e-kube-api-access-5k6r5\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:40.237622 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.237586    2581 generic.go:356] "Generic (PLEG): container finished" podID="79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e" containerID="1ed92a2896c8e4f1294cbe677f3bfe54fba4258981023ecab5073f011f043c18" exitCode=0
Sep 16 08:00:40.237838 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.237629    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"
Sep 16 08:00:40.237838 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.237670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd" event={"ID":"79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e","Type":"ContainerDied","Data":"1ed92a2896c8e4f1294cbe677f3bfe54fba4258981023ecab5073f011f043c18"}
Sep 16 08:00:40.237838 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.237727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd" event={"ID":"79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e","Type":"ContainerDied","Data":"f492fdc8e553877f62db8671a7872d3f336ec62d5f760636a757848f08cbe0a4"}
Sep 16 08:00:40.237838 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.237757    2581 scope.go:117] "RemoveContainer" containerID="1ed92a2896c8e4f1294cbe677f3bfe54fba4258981023ecab5073f011f043c18"
Sep 16 08:00:40.247171 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.247152    2581 scope.go:117] "RemoveContainer" containerID="1ed92a2896c8e4f1294cbe677f3bfe54fba4258981023ecab5073f011f043c18"
Sep 16 08:00:40.247448 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:00:40.247432    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed92a2896c8e4f1294cbe677f3bfe54fba4258981023ecab5073f011f043c18\": container with ID starting with 1ed92a2896c8e4f1294cbe677f3bfe54fba4258981023ecab5073f011f043c18 not found: ID does not exist" containerID="1ed92a2896c8e4f1294cbe677f3bfe54fba4258981023ecab5073f011f043c18"
Sep 16 08:00:40.247502 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.247455    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed92a2896c8e4f1294cbe677f3bfe54fba4258981023ecab5073f011f043c18"} err="failed to get container status \"1ed92a2896c8e4f1294cbe677f3bfe54fba4258981023ecab5073f011f043c18\": rpc error: code = NotFound desc = could not find container \"1ed92a2896c8e4f1294cbe677f3bfe54fba4258981023ecab5073f011f043c18\": container with ID starting with 1ed92a2896c8e4f1294cbe677f3bfe54fba4258981023ecab5073f011f043c18 not found: ID does not exist"
Sep 16 08:00:40.260136 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.260106    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"]
Sep 16 08:00:40.263810 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:40.263784    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4a243c-54575cbff8-ztwbd"]
Sep 16 08:00:41.685046 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:41.685015    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ead664e-13c8-41d8-aeb8-f1e3b78f8157" path="/var/lib/kubelet/pods/5ead664e-13c8-41d8-aeb8-f1e3b78f8157/volumes"
Sep 16 08:00:41.685447 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:41.685351    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e" path="/var/lib/kubelet/pods/79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e/volumes"
Sep 16 08:00:46.132024 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:46.131991    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"]
Sep 16 08:00:46.132470 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:46.132221    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l" podUID="3418d981-b15f-4ea2-a807-75b731ea0804" containerName="istio-proxy" containerID="cri-o://d75fb7b051388addb56aecd0ab5267825132491bd93cf634a24f4d0b0b650989" gracePeriod=30
Sep 16 08:00:46.303592 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:46.303550    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 08:00:46.303760 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:46.303641    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l" podUID="3418d981-b15f-4ea2-a807-75b731ea0804" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 08:00:47.705690 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:47.705629    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2"]
Sep 16 08:00:47.710186 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:47.710139    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fc27a5-g4tj2"]
Sep 16 08:00:49.683943 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:49.683910    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2a6933b-e889-4ae6-83d8-d512efc75536" path="/var/lib/kubelet/pods/f2a6933b-e889-4ae6-83d8-d512efc75536/volumes"
Sep 16 08:00:50.238204 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.238149    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58" podUID="06b891c9-f64a-4833-8bd0-df3d3482f956" containerName="main" containerID="cri-o://151f9f63a1558fdb5fff15e1a2f8c6c4d710ba79906e44e6b41653dcbf3049f3" gracePeriod=45
Sep 16 08:00:50.494637 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.494580    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 08:00:50.587140 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.587104    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-model-cache\") pod \"06b891c9-f64a-4833-8bd0-df3d3482f956\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") "
Sep 16 08:00:50.587323 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.587154    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-tmp-dir\") pod \"06b891c9-f64a-4833-8bd0-df3d3482f956\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") "
Sep 16 08:00:50.587323 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.587182    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-dshm\") pod \"06b891c9-f64a-4833-8bd0-df3d3482f956\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") "
Sep 16 08:00:50.587323 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.587247    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06b891c9-f64a-4833-8bd0-df3d3482f956-tls-certs\") pod \"06b891c9-f64a-4833-8bd0-df3d3482f956\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") "
Sep 16 08:00:50.587523 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.587345    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvx6g\" (UniqueName: \"kubernetes.io/projected/06b891c9-f64a-4833-8bd0-df3d3482f956-kube-api-access-lvx6g\") pod \"06b891c9-f64a-4833-8bd0-df3d3482f956\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") "
Sep 16 08:00:50.587523 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.587411    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-home\") pod \"06b891c9-f64a-4833-8bd0-df3d3482f956\" (UID: \"06b891c9-f64a-4833-8bd0-df3d3482f956\") "
Sep 16 08:00:50.587523 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.587422    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-model-cache" (OuterVolumeSpecName: "model-cache") pod "06b891c9-f64a-4833-8bd0-df3d3482f956" (UID: "06b891c9-f64a-4833-8bd0-df3d3482f956"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:50.587523 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.587478    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06b891c9-f64a-4833-8bd0-df3d3482f956" (UID: "06b891c9-f64a-4833-8bd0-df3d3482f956"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:50.587736 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.587708    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-home" (OuterVolumeSpecName: "home") pod "06b891c9-f64a-4833-8bd0-df3d3482f956" (UID: "06b891c9-f64a-4833-8bd0-df3d3482f956"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:50.587945 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.587844    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:50.587945 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.587888    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:50.587945 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.587907    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:50.590125 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.590101    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06b891c9-f64a-4833-8bd0-df3d3482f956-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06b891c9-f64a-4833-8bd0-df3d3482f956" (UID: "06b891c9-f64a-4833-8bd0-df3d3482f956"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 08:00:50.590261 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.590239    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-dshm" (OuterVolumeSpecName: "dshm") pod "06b891c9-f64a-4833-8bd0-df3d3482f956" (UID: "06b891c9-f64a-4833-8bd0-df3d3482f956"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:50.590325 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.590258    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b891c9-f64a-4833-8bd0-df3d3482f956-kube-api-access-lvx6g" (OuterVolumeSpecName: "kube-api-access-lvx6g") pod "06b891c9-f64a-4833-8bd0-df3d3482f956" (UID: "06b891c9-f64a-4833-8bd0-df3d3482f956"). InnerVolumeSpecName "kube-api-access-lvx6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:00:50.689221 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.689190    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06b891c9-f64a-4833-8bd0-df3d3482f956-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:50.689221 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.689217    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06b891c9-f64a-4833-8bd0-df3d3482f956-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:50.689221 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:50.689229    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvx6g\" (UniqueName: \"kubernetes.io/projected/06b891c9-f64a-4833-8bd0-df3d3482f956-kube-api-access-lvx6g\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:51.302648 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.302611    2581 generic.go:356] "Generic (PLEG): container finished" podID="06b891c9-f64a-4833-8bd0-df3d3482f956" containerID="151f9f63a1558fdb5fff15e1a2f8c6c4d710ba79906e44e6b41653dcbf3049f3" exitCode=0
Sep 16 08:00:51.302870 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.302659    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"
Sep 16 08:00:51.302870 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.302696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58" event={"ID":"06b891c9-f64a-4833-8bd0-df3d3482f956","Type":"ContainerDied","Data":"151f9f63a1558fdb5fff15e1a2f8c6c4d710ba79906e44e6b41653dcbf3049f3"}
Sep 16 08:00:51.302870 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.302750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58" event={"ID":"06b891c9-f64a-4833-8bd0-df3d3482f956","Type":"ContainerDied","Data":"ac0a07057ff9aea9224bc2135bc28422f654eb176243c63d296e224d774f3f5d"}
Sep 16 08:00:51.302870 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.302774    2581 scope.go:117] "RemoveContainer" containerID="151f9f63a1558fdb5fff15e1a2f8c6c4d710ba79906e44e6b41653dcbf3049f3"
Sep 16 08:00:51.305679 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.305646    2581 generic.go:356] "Generic (PLEG): container finished" podID="3418d981-b15f-4ea2-a807-75b731ea0804" containerID="d75fb7b051388addb56aecd0ab5267825132491bd93cf634a24f4d0b0b650989" exitCode=0
Sep 16 08:00:51.305921 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.305876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l" event={"ID":"3418d981-b15f-4ea2-a807-75b731ea0804","Type":"ContainerDied","Data":"d75fb7b051388addb56aecd0ab5267825132491bd93cf634a24f4d0b0b650989"}
Sep 16 08:00:51.314331 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.313814    2581 scope.go:117] "RemoveContainer" containerID="151f9f63a1558fdb5fff15e1a2f8c6c4d710ba79906e44e6b41653dcbf3049f3"
Sep 16 08:00:51.314612 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:00:51.314579    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151f9f63a1558fdb5fff15e1a2f8c6c4d710ba79906e44e6b41653dcbf3049f3\": container with ID starting with 151f9f63a1558fdb5fff15e1a2f8c6c4d710ba79906e44e6b41653dcbf3049f3 not found: ID does not exist" containerID="151f9f63a1558fdb5fff15e1a2f8c6c4d710ba79906e44e6b41653dcbf3049f3"
Sep 16 08:00:51.314691 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.314625    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151f9f63a1558fdb5fff15e1a2f8c6c4d710ba79906e44e6b41653dcbf3049f3"} err="failed to get container status \"151f9f63a1558fdb5fff15e1a2f8c6c4d710ba79906e44e6b41653dcbf3049f3\": rpc error: code = NotFound desc = could not find container \"151f9f63a1558fdb5fff15e1a2f8c6c4d710ba79906e44e6b41653dcbf3049f3\": container with ID starting with 151f9f63a1558fdb5fff15e1a2f8c6c4d710ba79906e44e6b41653dcbf3049f3 not found: ID does not exist"
Sep 16 08:00:51.314691 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.314680    2581 scope.go:117] "RemoveContainer" containerID="795f0b061a10bd49bbf7833f8f1dc62bd052786080320164024a9e86d5600d7d"
Sep 16 08:00:51.332209 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.332166    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"]
Sep 16 08:00:51.335512 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.335484    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/distinct-2-w4-fc27a5-kserve-6748dbf5db-kbv58"]
Sep 16 08:00:51.397728 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.397657    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 08:00:51.496962 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.496924    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-istio-envoy\") pod \"3418d981-b15f-4ea2-a807-75b731ea0804\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") "
Sep 16 08:00:51.497162 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.496983    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-credential-socket\") pod \"3418d981-b15f-4ea2-a807-75b731ea0804\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") "
Sep 16 08:00:51.497162 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-workload-certs\") pod \"3418d981-b15f-4ea2-a807-75b731ea0804\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") "
Sep 16 08:00:51.497162 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497060    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vk4v\" (UniqueName: \"kubernetes.io/projected/3418d981-b15f-4ea2-a807-75b731ea0804-kube-api-access-4vk4v\") pod \"3418d981-b15f-4ea2-a807-75b731ea0804\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") "
Sep 16 08:00:51.497162 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497084    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-workload-socket\") pod \"3418d981-b15f-4ea2-a807-75b731ea0804\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") "
Sep 16 08:00:51.497162 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497139    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3418d981-b15f-4ea2-a807-75b731ea0804-istio-podinfo\") pod \"3418d981-b15f-4ea2-a807-75b731ea0804\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") "
Sep 16 08:00:51.497428 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497232    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-istio-data\") pod \"3418d981-b15f-4ea2-a807-75b731ea0804\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") "
Sep 16 08:00:51.497428 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497270    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3418d981-b15f-4ea2-a807-75b731ea0804-istiod-ca-cert\") pod \"3418d981-b15f-4ea2-a807-75b731ea0804\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") "
Sep 16 08:00:51.497428 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497297    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3418d981-b15f-4ea2-a807-75b731ea0804-istio-token\") pod \"3418d981-b15f-4ea2-a807-75b731ea0804\" (UID: \"3418d981-b15f-4ea2-a807-75b731ea0804\") "
Sep 16 08:00:51.497428 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497294    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3418d981-b15f-4ea2-a807-75b731ea0804" (UID: "3418d981-b15f-4ea2-a807-75b731ea0804"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:51.497428 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497310    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3418d981-b15f-4ea2-a807-75b731ea0804" (UID: "3418d981-b15f-4ea2-a807-75b731ea0804"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:51.497428 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497390    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3418d981-b15f-4ea2-a807-75b731ea0804" (UID: "3418d981-b15f-4ea2-a807-75b731ea0804"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:51.497728 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497537    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-istio-data" (OuterVolumeSpecName: "istio-data") pod "3418d981-b15f-4ea2-a807-75b731ea0804" (UID: "3418d981-b15f-4ea2-a807-75b731ea0804"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:51.497728 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497604    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3418d981-b15f-4ea2-a807-75b731ea0804-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3418d981-b15f-4ea2-a807-75b731ea0804" (UID: "3418d981-b15f-4ea2-a807-75b731ea0804"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:00:51.497815 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497739    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:51.497815 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497756    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:51.497815 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497769    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3418d981-b15f-4ea2-a807-75b731ea0804-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:51.497815 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497782    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:51.497815 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.497795    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:51.499488 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.499453    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3418d981-b15f-4ea2-a807-75b731ea0804-istio-token" (OuterVolumeSpecName: "istio-token") pod "3418d981-b15f-4ea2-a807-75b731ea0804" (UID: "3418d981-b15f-4ea2-a807-75b731ea0804"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:00:51.499593 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.499518    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3418d981-b15f-4ea2-a807-75b731ea0804-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3418d981-b15f-4ea2-a807-75b731ea0804" (UID: "3418d981-b15f-4ea2-a807-75b731ea0804"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 08:00:51.499668 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.499643    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3418d981-b15f-4ea2-a807-75b731ea0804" (UID: "3418d981-b15f-4ea2-a807-75b731ea0804"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:51.499711 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.499675    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3418d981-b15f-4ea2-a807-75b731ea0804-kube-api-access-4vk4v" (OuterVolumeSpecName: "kube-api-access-4vk4v") pod "3418d981-b15f-4ea2-a807-75b731ea0804" (UID: "3418d981-b15f-4ea2-a807-75b731ea0804"). InnerVolumeSpecName "kube-api-access-4vk4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:00:51.599020 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.598977    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3418d981-b15f-4ea2-a807-75b731ea0804-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:51.599020 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.599012    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3418d981-b15f-4ea2-a807-75b731ea0804-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:51.599020 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.599022    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vk4v\" (UniqueName: \"kubernetes.io/projected/3418d981-b15f-4ea2-a807-75b731ea0804-kube-api-access-4vk4v\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:51.599020 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.599030    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3418d981-b15f-4ea2-a807-75b731ea0804-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:51.688746 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:51.688705    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06b891c9-f64a-4833-8bd0-df3d3482f956" path="/var/lib/kubelet/pods/06b891c9-f64a-4833-8bd0-df3d3482f956/volumes"
Sep 16 08:00:52.315840 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:52.315777    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l" event={"ID":"3418d981-b15f-4ea2-a807-75b731ea0804","Type":"ContainerDied","Data":"a199be26fe5f91364c82f181e9fad5d336d0eec39b4eb536aa66a342923196a1"}
Sep 16 08:00:52.315840 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:52.315801    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"
Sep 16 08:00:52.316390 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:52.315881    2581 scope.go:117] "RemoveContainer" containerID="d75fb7b051388addb56aecd0ab5267825132491bd93cf634a24f4d0b0b650989"
Sep 16 08:00:52.337789 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:52.337757    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"]
Sep 16 08:00:52.341601 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:52.341572    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4a243c-openshift-default-d45555ffb-5zk9l"]
Sep 16 08:00:53.690746 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:53.690706    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3418d981-b15f-4ea2-a807-75b731ea0804" path="/var/lib/kubelet/pods/3418d981-b15f-4ea2-a807-75b731ea0804/volumes"
Sep 16 08:00:54.083862 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:54.083726    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"]
Sep 16 08:00:54.084072 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:54.084042    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr" podUID="8999f450-ddca-4885-af15-c203abdf47d7" containerName="istio-proxy" containerID="cri-o://b08d1465f1e10724dcb4c93dc687f936aa2d0d48e48254f12185ad98a670f0bb" gracePeriod=30
Sep 16 08:00:57.324647 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.324608    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"]
Sep 16 08:00:57.325091 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.324880    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n" podUID="881465a0-eb62-4314-b5d2-eb180a5d600d" containerName="payload-processing" containerID="cri-o://01304911a85ccd4813ed79daac63fc513bd0882985f9da2e2bc5be4e9c1def34" gracePeriod=30
Sep 16 08:00:57.331111 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.331081    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"]
Sep 16 08:00:57.331328 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.331302    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx" podUID="41a6be09-4aaf-42c7-a00c-33cb8bcb3f45" containerName="payload-pre-processing" containerID="cri-o://210706d31e345bdf41f7b3373a83b20b530316bcc478b138c4b2ba725c88137b" gracePeriod=30
Sep 16 08:00:57.643995 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.643973    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"
Sep 16 08:00:57.646851 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.646812    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"
Sep 16 08:00:57.655723 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.655704    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a6be09-4aaf-42c7-a00c-33cb8bcb3f45-plugins-config-volume\") pod \"41a6be09-4aaf-42c7-a00c-33cb8bcb3f45\" (UID: \"41a6be09-4aaf-42c7-a00c-33cb8bcb3f45\") "
Sep 16 08:00:57.655811 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.655776    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjsjm\" (UniqueName: \"kubernetes.io/projected/881465a0-eb62-4314-b5d2-eb180a5d600d-kube-api-access-xjsjm\") pod \"881465a0-eb62-4314-b5d2-eb180a5d600d\" (UID: \"881465a0-eb62-4314-b5d2-eb180a5d600d\") "
Sep 16 08:00:57.655879 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.655841    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881465a0-eb62-4314-b5d2-eb180a5d600d-plugins-config-volume\") pod \"881465a0-eb62-4314-b5d2-eb180a5d600d\" (UID: \"881465a0-eb62-4314-b5d2-eb180a5d600d\") "
Sep 16 08:00:57.655879 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.655859    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6glld\" (UniqueName: \"kubernetes.io/projected/41a6be09-4aaf-42c7-a00c-33cb8bcb3f45-kube-api-access-6glld\") pod \"41a6be09-4aaf-42c7-a00c-33cb8bcb3f45\" (UID: \"41a6be09-4aaf-42c7-a00c-33cb8bcb3f45\") "
Sep 16 08:00:57.656117 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.656090    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a6be09-4aaf-42c7-a00c-33cb8bcb3f45-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41a6be09-4aaf-42c7-a00c-33cb8bcb3f45" (UID: "41a6be09-4aaf-42c7-a00c-33cb8bcb3f45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:00:57.656240 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.656220    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881465a0-eb62-4314-b5d2-eb180a5d600d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "881465a0-eb62-4314-b5d2-eb180a5d600d" (UID: "881465a0-eb62-4314-b5d2-eb180a5d600d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:00:57.657926 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.657900    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881465a0-eb62-4314-b5d2-eb180a5d600d-kube-api-access-xjsjm" (OuterVolumeSpecName: "kube-api-access-xjsjm") pod "881465a0-eb62-4314-b5d2-eb180a5d600d" (UID: "881465a0-eb62-4314-b5d2-eb180a5d600d"). InnerVolumeSpecName "kube-api-access-xjsjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:00:57.658007 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.657954    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a6be09-4aaf-42c7-a00c-33cb8bcb3f45-kube-api-access-6glld" (OuterVolumeSpecName: "kube-api-access-6glld") pod "41a6be09-4aaf-42c7-a00c-33cb8bcb3f45" (UID: "41a6be09-4aaf-42c7-a00c-33cb8bcb3f45"). InnerVolumeSpecName "kube-api-access-6glld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:00:57.756887 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.756853    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881465a0-eb62-4314-b5d2-eb180a5d600d-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:57.756887 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.756888    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6glld\" (UniqueName: \"kubernetes.io/projected/41a6be09-4aaf-42c7-a00c-33cb8bcb3f45-kube-api-access-6glld\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:57.757068 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.756898    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a6be09-4aaf-42c7-a00c-33cb8bcb3f45-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:57.757068 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:57.756907    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjsjm\" (UniqueName: \"kubernetes.io/projected/881465a0-eb62-4314-b5d2-eb180a5d600d-kube-api-access-xjsjm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:58.354372 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.354343    2581 generic.go:356] "Generic (PLEG): container finished" podID="41a6be09-4aaf-42c7-a00c-33cb8bcb3f45" containerID="210706d31e345bdf41f7b3373a83b20b530316bcc478b138c4b2ba725c88137b" exitCode=0
Sep 16 08:00:58.354791 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.354396    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"
Sep 16 08:00:58.354791 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.354416    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx" event={"ID":"41a6be09-4aaf-42c7-a00c-33cb8bcb3f45","Type":"ContainerDied","Data":"210706d31e345bdf41f7b3373a83b20b530316bcc478b138c4b2ba725c88137b"}
Sep 16 08:00:58.354791 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.354465    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx" event={"ID":"41a6be09-4aaf-42c7-a00c-33cb8bcb3f45","Type":"ContainerDied","Data":"2f9df9b5313af797e1845d0a675452b5e86ff1fbaad452c185c3050b1fabd007"}
Sep 16 08:00:58.354791 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.354488    2581 scope.go:117] "RemoveContainer" containerID="210706d31e345bdf41f7b3373a83b20b530316bcc478b138c4b2ba725c88137b"
Sep 16 08:00:58.355966 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.355944    2581 generic.go:356] "Generic (PLEG): container finished" podID="881465a0-eb62-4314-b5d2-eb180a5d600d" containerID="01304911a85ccd4813ed79daac63fc513bd0882985f9da2e2bc5be4e9c1def34" exitCode=0
Sep 16 08:00:58.356055 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.355986    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"
Sep 16 08:00:58.356055 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.356024    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n" event={"ID":"881465a0-eb62-4314-b5d2-eb180a5d600d","Type":"ContainerDied","Data":"01304911a85ccd4813ed79daac63fc513bd0882985f9da2e2bc5be4e9c1def34"}
Sep 16 08:00:58.356144 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.356075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n" event={"ID":"881465a0-eb62-4314-b5d2-eb180a5d600d","Type":"ContainerDied","Data":"96a88a6fb832b21c0956835926a6a511c6c973d6ffa086e9bf85131ee58e1a65"}
Sep 16 08:00:58.363595 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.363576    2581 scope.go:117] "RemoveContainer" containerID="210706d31e345bdf41f7b3373a83b20b530316bcc478b138c4b2ba725c88137b"
Sep 16 08:00:58.363866 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:00:58.363845    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"210706d31e345bdf41f7b3373a83b20b530316bcc478b138c4b2ba725c88137b\": container with ID starting with 210706d31e345bdf41f7b3373a83b20b530316bcc478b138c4b2ba725c88137b not found: ID does not exist" containerID="210706d31e345bdf41f7b3373a83b20b530316bcc478b138c4b2ba725c88137b"
Sep 16 08:00:58.363957 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.363877    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"210706d31e345bdf41f7b3373a83b20b530316bcc478b138c4b2ba725c88137b"} err="failed to get container status \"210706d31e345bdf41f7b3373a83b20b530316bcc478b138c4b2ba725c88137b\": rpc error: code = NotFound desc = could not find container \"210706d31e345bdf41f7b3373a83b20b530316bcc478b138c4b2ba725c88137b\": container with ID starting with 210706d31e345bdf41f7b3373a83b20b530316bcc478b138c4b2ba725c88137b not found: ID does not exist"
Sep 16 08:00:58.363957 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.363898    2581 scope.go:117] "RemoveContainer" containerID="01304911a85ccd4813ed79daac63fc513bd0882985f9da2e2bc5be4e9c1def34"
Sep 16 08:00:58.371603 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.371588    2581 scope.go:117] "RemoveContainer" containerID="01304911a85ccd4813ed79daac63fc513bd0882985f9da2e2bc5be4e9c1def34"
Sep 16 08:00:58.371849 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:00:58.371812    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01304911a85ccd4813ed79daac63fc513bd0882985f9da2e2bc5be4e9c1def34\": container with ID starting with 01304911a85ccd4813ed79daac63fc513bd0882985f9da2e2bc5be4e9c1def34 not found: ID does not exist" containerID="01304911a85ccd4813ed79daac63fc513bd0882985f9da2e2bc5be4e9c1def34"
Sep 16 08:00:58.371922 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.371860    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01304911a85ccd4813ed79daac63fc513bd0882985f9da2e2bc5be4e9c1def34"} err="failed to get container status \"01304911a85ccd4813ed79daac63fc513bd0882985f9da2e2bc5be4e9c1def34\": rpc error: code = NotFound desc = could not find container \"01304911a85ccd4813ed79daac63fc513bd0882985f9da2e2bc5be4e9c1def34\": container with ID starting with 01304911a85ccd4813ed79daac63fc513bd0882985f9da2e2bc5be4e9c1def34 not found: ID does not exist"
Sep 16 08:00:58.377045 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.377024    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"]
Sep 16 08:00:58.378993 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.378970    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-984c1a-7949b878b-t9nnx"]
Sep 16 08:00:58.388058 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.388028    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"]
Sep 16 08:00:58.395874 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:58.393693    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-984c1a-785896b699-d2p5n"]
Sep 16 08:00:59.342924 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.342899    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 08:00:59.368198 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.368119    2581 generic.go:356] "Generic (PLEG): container finished" podID="8999f450-ddca-4885-af15-c203abdf47d7" containerID="b08d1465f1e10724dcb4c93dc687f936aa2d0d48e48254f12185ad98a670f0bb" exitCode=0
Sep 16 08:00:59.368198 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.368161    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr" event={"ID":"8999f450-ddca-4885-af15-c203abdf47d7","Type":"ContainerDied","Data":"b08d1465f1e10724dcb4c93dc687f936aa2d0d48e48254f12185ad98a670f0bb"}
Sep 16 08:00:59.368198 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.368170    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"
Sep 16 08:00:59.368629 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.368234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr" event={"ID":"8999f450-ddca-4885-af15-c203abdf47d7","Type":"ContainerDied","Data":"5bea0f225ee72bb262df5085f2537843603f781cd3883241dba946707ef1964a"}
Sep 16 08:00:59.368629 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.368258    2581 scope.go:117] "RemoveContainer" containerID="b08d1465f1e10724dcb4c93dc687f936aa2d0d48e48254f12185ad98a670f0bb"
Sep 16 08:00:59.373564 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.373538    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-workload-socket\") pod \"8999f450-ddca-4885-af15-c203abdf47d7\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") "
Sep 16 08:00:59.373712 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.373584    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8999f450-ddca-4885-af15-c203abdf47d7-istiod-ca-cert\") pod \"8999f450-ddca-4885-af15-c203abdf47d7\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") "
Sep 16 08:00:59.373712 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.373611    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-workload-certs\") pod \"8999f450-ddca-4885-af15-c203abdf47d7\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") "
Sep 16 08:00:59.373712 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.373643    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-istio-envoy\") pod \"8999f450-ddca-4885-af15-c203abdf47d7\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") "
Sep 16 08:00:59.373712 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.373672    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8999f450-ddca-4885-af15-c203abdf47d7-istio-token\") pod \"8999f450-ddca-4885-af15-c203abdf47d7\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") "
Sep 16 08:00:59.373712 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.373708    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8999f450-ddca-4885-af15-c203abdf47d7-istio-podinfo\") pod \"8999f450-ddca-4885-af15-c203abdf47d7\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") "
Sep 16 08:00:59.374025 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.373755    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-istio-data\") pod \"8999f450-ddca-4885-af15-c203abdf47d7\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") "
Sep 16 08:00:59.374025 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.373794    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz4wf\" (UniqueName: \"kubernetes.io/projected/8999f450-ddca-4885-af15-c203abdf47d7-kube-api-access-sz4wf\") pod \"8999f450-ddca-4885-af15-c203abdf47d7\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") "
Sep 16 08:00:59.374025 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.373794    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8999f450-ddca-4885-af15-c203abdf47d7" (UID: "8999f450-ddca-4885-af15-c203abdf47d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:59.374025 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.373875    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-credential-socket\") pod \"8999f450-ddca-4885-af15-c203abdf47d7\" (UID: \"8999f450-ddca-4885-af15-c203abdf47d7\") "
Sep 16 08:00:59.374025 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.373953    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8999f450-ddca-4885-af15-c203abdf47d7" (UID: "8999f450-ddca-4885-af15-c203abdf47d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:59.374275 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.374195    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8999f450-ddca-4885-af15-c203abdf47d7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8999f450-ddca-4885-af15-c203abdf47d7" (UID: "8999f450-ddca-4885-af15-c203abdf47d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:00:59.374275 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.374258    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-istio-data" (OuterVolumeSpecName: "istio-data") pod "8999f450-ddca-4885-af15-c203abdf47d7" (UID: "8999f450-ddca-4885-af15-c203abdf47d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:59.374386 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.374327    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:59.374386 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.374370    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:59.374586 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.374565    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8999f450-ddca-4885-af15-c203abdf47d7" (UID: "8999f450-ddca-4885-af15-c203abdf47d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:59.376721 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.376694    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8999f450-ddca-4885-af15-c203abdf47d7-kube-api-access-sz4wf" (OuterVolumeSpecName: "kube-api-access-sz4wf") pod "8999f450-ddca-4885-af15-c203abdf47d7" (UID: "8999f450-ddca-4885-af15-c203abdf47d7"). InnerVolumeSpecName "kube-api-access-sz4wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:00:59.377276 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.377250    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8999f450-ddca-4885-af15-c203abdf47d7-istio-token" (OuterVolumeSpecName: "istio-token") pod "8999f450-ddca-4885-af15-c203abdf47d7" (UID: "8999f450-ddca-4885-af15-c203abdf47d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:00:59.377394 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.377292    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8999f450-ddca-4885-af15-c203abdf47d7" (UID: "8999f450-ddca-4885-af15-c203abdf47d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:00:59.377458 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.377437    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8999f450-ddca-4885-af15-c203abdf47d7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8999f450-ddca-4885-af15-c203abdf47d7" (UID: "8999f450-ddca-4885-af15-c203abdf47d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 08:00:59.381221 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.381196    2581 scope.go:117] "RemoveContainer" containerID="cef60180dd544efa6550d3548da28f32d6165b814f2253ea03045276f6821750"
Sep 16 08:00:59.395936 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.395912    2581 scope.go:117] "RemoveContainer" containerID="b08d1465f1e10724dcb4c93dc687f936aa2d0d48e48254f12185ad98a670f0bb"
Sep 16 08:00:59.396296 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:00:59.396273    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08d1465f1e10724dcb4c93dc687f936aa2d0d48e48254f12185ad98a670f0bb\": container with ID starting with b08d1465f1e10724dcb4c93dc687f936aa2d0d48e48254f12185ad98a670f0bb not found: ID does not exist" containerID="b08d1465f1e10724dcb4c93dc687f936aa2d0d48e48254f12185ad98a670f0bb"
Sep 16 08:00:59.396364 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.396305    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08d1465f1e10724dcb4c93dc687f936aa2d0d48e48254f12185ad98a670f0bb"} err="failed to get container status \"b08d1465f1e10724dcb4c93dc687f936aa2d0d48e48254f12185ad98a670f0bb\": rpc error: code = NotFound desc = could not find container \"b08d1465f1e10724dcb4c93dc687f936aa2d0d48e48254f12185ad98a670f0bb\": container with ID starting with b08d1465f1e10724dcb4c93dc687f936aa2d0d48e48254f12185ad98a670f0bb not found: ID does not exist"
Sep 16 08:00:59.396364 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.396325    2581 scope.go:117] "RemoveContainer" containerID="cef60180dd544efa6550d3548da28f32d6165b814f2253ea03045276f6821750"
Sep 16 08:00:59.396612 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:00:59.396584    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef60180dd544efa6550d3548da28f32d6165b814f2253ea03045276f6821750\": container with ID starting with cef60180dd544efa6550d3548da28f32d6165b814f2253ea03045276f6821750 not found: ID does not exist" containerID="cef60180dd544efa6550d3548da28f32d6165b814f2253ea03045276f6821750"
Sep 16 08:00:59.396669 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.396621    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef60180dd544efa6550d3548da28f32d6165b814f2253ea03045276f6821750"} err="failed to get container status \"cef60180dd544efa6550d3548da28f32d6165b814f2253ea03045276f6821750\": rpc error: code = NotFound desc = could not find container \"cef60180dd544efa6550d3548da28f32d6165b814f2253ea03045276f6821750\": container with ID starting with cef60180dd544efa6550d3548da28f32d6165b814f2253ea03045276f6821750 not found: ID does not exist"
Sep 16 08:00:59.474916 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.474881    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8999f450-ddca-4885-af15-c203abdf47d7-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:59.474916 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.474911    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8999f450-ddca-4885-af15-c203abdf47d7-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:59.475122 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.474923    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:59.475122 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.474937    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz4wf\" (UniqueName: \"kubernetes.io/projected/8999f450-ddca-4885-af15-c203abdf47d7-kube-api-access-sz4wf\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:59.475122 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.474950    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:59.475122 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.474960    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8999f450-ddca-4885-af15-c203abdf47d7-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:59.475122 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.474971    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8999f450-ddca-4885-af15-c203abdf47d7-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:00:59.683878 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.683853    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41a6be09-4aaf-42c7-a00c-33cb8bcb3f45" path="/var/lib/kubelet/pods/41a6be09-4aaf-42c7-a00c-33cb8bcb3f45/volumes"
Sep 16 08:00:59.684233 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.684219    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="881465a0-eb62-4314-b5d2-eb180a5d600d" path="/var/lib/kubelet/pods/881465a0-eb62-4314-b5d2-eb180a5d600d/volumes"
Sep 16 08:00:59.689594 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.689569    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"]
Sep 16 08:00:59.696997 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.696973    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fc27a5-openshift-default-f4fb5d448-z2ffr"]
Sep 16 08:00:59.855098 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:00:59.855065    2581 scope.go:117] "RemoveContainer" containerID="528f7b5aa0b2f6acdf944d8eebd773ba5f3e46de58a15dce0a7320bc9a1d9d0e"
Sep 16 08:01:01.684350 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:01.684306    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8999f450-ddca-4885-af15-c203abdf47d7" path="/var/lib/kubelet/pods/8999f450-ddca-4885-af15-c203abdf47d7/volumes"
Sep 16 08:01:04.339813 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:04.339778    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"]
Sep 16 08:01:04.345940 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:04.345901    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2" podUID="7888826b-cd84-4cc6-b808-e18a75041202" containerName="istio-proxy" containerID="cri-o://4d564fa0976c0b37912f2c83d28e27b0179d5a8b02092900ee5fe0fb4a196640" gracePeriod=30
Sep 16 08:01:09.602152 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.602125    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 08:01:09.673538 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.673500    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-workload-socket\") pod \"7888826b-cd84-4cc6-b808-e18a75041202\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") "
Sep 16 08:01:09.673722 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.673572    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-workload-certs\") pod \"7888826b-cd84-4cc6-b808-e18a75041202\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") "
Sep 16 08:01:09.673722 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.673606    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8mbh\" (UniqueName: \"kubernetes.io/projected/7888826b-cd84-4cc6-b808-e18a75041202-kube-api-access-p8mbh\") pod \"7888826b-cd84-4cc6-b808-e18a75041202\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") "
Sep 16 08:01:09.673722 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.673628    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-credential-socket\") pod \"7888826b-cd84-4cc6-b808-e18a75041202\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") "
Sep 16 08:01:09.673722 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.673646    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-istio-envoy\") pod \"7888826b-cd84-4cc6-b808-e18a75041202\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") "
Sep 16 08:01:09.673722 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.673662    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7888826b-cd84-4cc6-b808-e18a75041202-istiod-ca-cert\") pod \"7888826b-cd84-4cc6-b808-e18a75041202\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") "
Sep 16 08:01:09.674028 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.673731    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7888826b-cd84-4cc6-b808-e18a75041202-istio-token\") pod \"7888826b-cd84-4cc6-b808-e18a75041202\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") "
Sep 16 08:01:09.674028 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.673788    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7888826b-cd84-4cc6-b808-e18a75041202-istio-podinfo\") pod \"7888826b-cd84-4cc6-b808-e18a75041202\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") "
Sep 16 08:01:09.674028 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.673853    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-istio-data\") pod \"7888826b-cd84-4cc6-b808-e18a75041202\" (UID: \"7888826b-cd84-4cc6-b808-e18a75041202\") "
Sep 16 08:01:09.674028 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.673853    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7888826b-cd84-4cc6-b808-e18a75041202" (UID: "7888826b-cd84-4cc6-b808-e18a75041202"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:01:09.674028 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.673938    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7888826b-cd84-4cc6-b808-e18a75041202" (UID: "7888826b-cd84-4cc6-b808-e18a75041202"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:01:09.674272 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.674051    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7888826b-cd84-4cc6-b808-e18a75041202" (UID: "7888826b-cd84-4cc6-b808-e18a75041202"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:01:09.674272 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.674152    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7888826b-cd84-4cc6-b808-e18a75041202-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7888826b-cd84-4cc6-b808-e18a75041202" (UID: "7888826b-cd84-4cc6-b808-e18a75041202"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:01:09.674272 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.674161    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-istio-data" (OuterVolumeSpecName: "istio-data") pod "7888826b-cd84-4cc6-b808-e18a75041202" (UID: "7888826b-cd84-4cc6-b808-e18a75041202"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:01:09.674272 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.674245    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:09.674272 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.674266    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:09.674535 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.674283    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:09.674535 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.674298    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:09.674535 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.674313    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7888826b-cd84-4cc6-b808-e18a75041202-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:09.676017 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.675988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7888826b-cd84-4cc6-b808-e18a75041202-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7888826b-cd84-4cc6-b808-e18a75041202" (UID: "7888826b-cd84-4cc6-b808-e18a75041202"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 08:01:09.676017 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.676006    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7888826b-cd84-4cc6-b808-e18a75041202-istio-token" (OuterVolumeSpecName: "istio-token") pod "7888826b-cd84-4cc6-b808-e18a75041202" (UID: "7888826b-cd84-4cc6-b808-e18a75041202"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:01:09.676164 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.676075    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7888826b-cd84-4cc6-b808-e18a75041202" (UID: "7888826b-cd84-4cc6-b808-e18a75041202"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:01:09.676164 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.676147    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7888826b-cd84-4cc6-b808-e18a75041202-kube-api-access-p8mbh" (OuterVolumeSpecName: "kube-api-access-p8mbh") pod "7888826b-cd84-4cc6-b808-e18a75041202" (UID: "7888826b-cd84-4cc6-b808-e18a75041202"). InnerVolumeSpecName "kube-api-access-p8mbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:01:09.774873 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.774800    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8mbh\" (UniqueName: \"kubernetes.io/projected/7888826b-cd84-4cc6-b808-e18a75041202-kube-api-access-p8mbh\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:09.774873 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.774868    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7888826b-cd84-4cc6-b808-e18a75041202-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:09.774873 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.774880    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7888826b-cd84-4cc6-b808-e18a75041202-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:09.775126 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:09.774888    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7888826b-cd84-4cc6-b808-e18a75041202-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:10.424435 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:10.424392    2581 generic.go:356] "Generic (PLEG): container finished" podID="7888826b-cd84-4cc6-b808-e18a75041202" containerID="4d564fa0976c0b37912f2c83d28e27b0179d5a8b02092900ee5fe0fb4a196640" exitCode=0
Sep 16 08:01:10.424591 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:10.424459    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"
Sep 16 08:01:10.424591 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:10.424473    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2" event={"ID":"7888826b-cd84-4cc6-b808-e18a75041202","Type":"ContainerDied","Data":"4d564fa0976c0b37912f2c83d28e27b0179d5a8b02092900ee5fe0fb4a196640"}
Sep 16 08:01:10.424591 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:10.424533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2" event={"ID":"7888826b-cd84-4cc6-b808-e18a75041202","Type":"ContainerDied","Data":"e02654bb3b737c58219c3feca4f238c67b3a36c1e034f2a6464bf633f333dd61"}
Sep 16 08:01:10.424591 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:10.424555    2581 scope.go:117] "RemoveContainer" containerID="4d564fa0976c0b37912f2c83d28e27b0179d5a8b02092900ee5fe0fb4a196640"
Sep 16 08:01:10.433652 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:10.433627    2581 scope.go:117] "RemoveContainer" containerID="839bfe8c3df14623e30bf98544b74d8fb604aae31e857f340a19f718033837de"
Sep 16 08:01:10.443403 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:10.443374    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"]
Sep 16 08:01:10.443489 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:10.443383    2581 scope.go:117] "RemoveContainer" containerID="4d564fa0976c0b37912f2c83d28e27b0179d5a8b02092900ee5fe0fb4a196640"
Sep 16 08:01:10.443782 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:01:10.443761    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d564fa0976c0b37912f2c83d28e27b0179d5a8b02092900ee5fe0fb4a196640\": container with ID starting with 4d564fa0976c0b37912f2c83d28e27b0179d5a8b02092900ee5fe0fb4a196640 not found: ID does not exist" containerID="4d564fa0976c0b37912f2c83d28e27b0179d5a8b02092900ee5fe0fb4a196640"
Sep 16 08:01:10.443876 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:10.443795    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d564fa0976c0b37912f2c83d28e27b0179d5a8b02092900ee5fe0fb4a196640"} err="failed to get container status \"4d564fa0976c0b37912f2c83d28e27b0179d5a8b02092900ee5fe0fb4a196640\": rpc error: code = NotFound desc = could not find container \"4d564fa0976c0b37912f2c83d28e27b0179d5a8b02092900ee5fe0fb4a196640\": container with ID starting with 4d564fa0976c0b37912f2c83d28e27b0179d5a8b02092900ee5fe0fb4a196640 not found: ID does not exist"
Sep 16 08:01:10.444002 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:10.443972    2581 scope.go:117] "RemoveContainer" containerID="839bfe8c3df14623e30bf98544b74d8fb604aae31e857f340a19f718033837de"
Sep 16 08:01:10.444438 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:01:10.444392    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"839bfe8c3df14623e30bf98544b74d8fb604aae31e857f340a19f718033837de\": container with ID starting with 839bfe8c3df14623e30bf98544b74d8fb604aae31e857f340a19f718033837de not found: ID does not exist" containerID="839bfe8c3df14623e30bf98544b74d8fb604aae31e857f340a19f718033837de"
Sep 16 08:01:10.444550 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:10.444431    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"839bfe8c3df14623e30bf98544b74d8fb604aae31e857f340a19f718033837de"} err="failed to get container status \"839bfe8c3df14623e30bf98544b74d8fb604aae31e857f340a19f718033837de\": rpc error: code = NotFound desc = could not find container \"839bfe8c3df14623e30bf98544b74d8fb604aae31e857f340a19f718033837de\": container with ID starting with 839bfe8c3df14623e30bf98544b74d8fb604aae31e857f340a19f718033837de not found: ID does not exist"
Sep 16 08:01:10.446655 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:10.446634    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-984c1a-openshift-default-7bb5887b68-x4nw2"]
Sep 16 08:01:11.684131 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:11.684097    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7888826b-cd84-4cc6-b808-e18a75041202" path="/var/lib/kubelet/pods/7888826b-cd84-4cc6-b808-e18a75041202/volumes"
Sep 16 08:01:37.200436 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:37.200394    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"]
Sep 16 08:01:38.171967 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:38.171933    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"]
Sep 16 08:01:52.220128 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.220071    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb" podUID="328dfdf4-6723-4d54-8902-2d217f6be6a4" containerName="main" containerID="cri-o://c6c9d0d80c975f01d5f983589aabc21a66640200b43144b3faac568e21d46f32" gracePeriod=45
Sep 16 08:01:52.472937 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.472883    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 08:01:52.592441 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.592410    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qcb9\" (UniqueName: \"kubernetes.io/projected/328dfdf4-6723-4d54-8902-2d217f6be6a4-kube-api-access-8qcb9\") pod \"328dfdf4-6723-4d54-8902-2d217f6be6a4\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") "
Sep 16 08:01:52.592627 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.592453    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-dshm\") pod \"328dfdf4-6723-4d54-8902-2d217f6be6a4\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") "
Sep 16 08:01:52.592627 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.592478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-home\") pod \"328dfdf4-6723-4d54-8902-2d217f6be6a4\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") "
Sep 16 08:01:52.592627 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.592551    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328dfdf4-6723-4d54-8902-2d217f6be6a4-tls-certs\") pod \"328dfdf4-6723-4d54-8902-2d217f6be6a4\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") "
Sep 16 08:01:52.592627 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.592603    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-model-cache\") pod \"328dfdf4-6723-4d54-8902-2d217f6be6a4\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") "
Sep 16 08:01:52.592880 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.592742    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-home" (OuterVolumeSpecName: "home") pod "328dfdf4-6723-4d54-8902-2d217f6be6a4" (UID: "328dfdf4-6723-4d54-8902-2d217f6be6a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:01:52.592880 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.592868    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-tmp-dir\") pod \"328dfdf4-6723-4d54-8902-2d217f6be6a4\" (UID: \"328dfdf4-6723-4d54-8902-2d217f6be6a4\") "
Sep 16 08:01:52.592984 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.592860    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "328dfdf4-6723-4d54-8902-2d217f6be6a4" (UID: "328dfdf4-6723-4d54-8902-2d217f6be6a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:01:52.592984 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.592921    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "328dfdf4-6723-4d54-8902-2d217f6be6a4" (UID: "328dfdf4-6723-4d54-8902-2d217f6be6a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:01:52.593144 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.593129    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:52.593194 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.593149    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:52.593194 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.593157    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:52.594776 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.594745    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328dfdf4-6723-4d54-8902-2d217f6be6a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "328dfdf4-6723-4d54-8902-2d217f6be6a4" (UID: "328dfdf4-6723-4d54-8902-2d217f6be6a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 08:01:52.595006 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.594986    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-dshm" (OuterVolumeSpecName: "dshm") pod "328dfdf4-6723-4d54-8902-2d217f6be6a4" (UID: "328dfdf4-6723-4d54-8902-2d217f6be6a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:01:52.595056 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.595000    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328dfdf4-6723-4d54-8902-2d217f6be6a4-kube-api-access-8qcb9" (OuterVolumeSpecName: "kube-api-access-8qcb9") pod "328dfdf4-6723-4d54-8902-2d217f6be6a4" (UID: "328dfdf4-6723-4d54-8902-2d217f6be6a4"). InnerVolumeSpecName "kube-api-access-8qcb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:01:52.641533 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.641507    2581 generic.go:356] "Generic (PLEG): container finished" podID="328dfdf4-6723-4d54-8902-2d217f6be6a4" containerID="c6c9d0d80c975f01d5f983589aabc21a66640200b43144b3faac568e21d46f32" exitCode=0
Sep 16 08:01:52.641662 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.641537    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb" event={"ID":"328dfdf4-6723-4d54-8902-2d217f6be6a4","Type":"ContainerDied","Data":"c6c9d0d80c975f01d5f983589aabc21a66640200b43144b3faac568e21d46f32"}
Sep 16 08:01:52.641662 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.641558    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"
Sep 16 08:01:52.641662 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.641577    2581 scope.go:117] "RemoveContainer" containerID="c6c9d0d80c975f01d5f983589aabc21a66640200b43144b3faac568e21d46f32"
Sep 16 08:01:52.641854 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.641568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb" event={"ID":"328dfdf4-6723-4d54-8902-2d217f6be6a4","Type":"ContainerDied","Data":"feba76be649b3f50738c92105ccff74b3e1fef02ec614bf8a83761bb36fa393b"}
Sep 16 08:01:52.650087 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.650071    2581 scope.go:117] "RemoveContainer" containerID="c6c9d0d80c975f01d5f983589aabc21a66640200b43144b3faac568e21d46f32"
Sep 16 08:01:52.650331 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:01:52.650315    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6c9d0d80c975f01d5f983589aabc21a66640200b43144b3faac568e21d46f32\": container with ID starting with c6c9d0d80c975f01d5f983589aabc21a66640200b43144b3faac568e21d46f32 not found: ID does not exist" containerID="c6c9d0d80c975f01d5f983589aabc21a66640200b43144b3faac568e21d46f32"
Sep 16 08:01:52.650377 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.650339    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6c9d0d80c975f01d5f983589aabc21a66640200b43144b3faac568e21d46f32"} err="failed to get container status \"c6c9d0d80c975f01d5f983589aabc21a66640200b43144b3faac568e21d46f32\": rpc error: code = NotFound desc = could not find container \"c6c9d0d80c975f01d5f983589aabc21a66640200b43144b3faac568e21d46f32\": container with ID starting with c6c9d0d80c975f01d5f983589aabc21a66640200b43144b3faac568e21d46f32 not found: ID does not exist"
Sep 16 08:01:52.662458 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.662438    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"]
Sep 16 08:01:52.668518 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.668497    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/auth-test-model-3616cc-kserve-75b86b89f9-8mrlb"]
Sep 16 08:01:52.694243 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.694224    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328dfdf4-6723-4d54-8902-2d217f6be6a4-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:52.694330 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.694247    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qcb9\" (UniqueName: \"kubernetes.io/projected/328dfdf4-6723-4d54-8902-2d217f6be6a4-kube-api-access-8qcb9\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:52.694330 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:52.694262    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328dfdf4-6723-4d54-8902-2d217f6be6a4-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:53.190900 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.190863    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf" podUID="c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2" containerName="main" containerID="cri-o://287b7c2f77175dffd8b7aa16a5ab7713040d115c4ec0194019f4bc9d9a727625" gracePeriod=45
Sep 16 08:01:53.449401 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.449336    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 08:01:53.602709 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.602671    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z24md\" (UniqueName: \"kubernetes.io/projected/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-kube-api-access-z24md\") pod \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") "
Sep 16 08:01:53.602709 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.602713    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-dshm\") pod \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") "
Sep 16 08:01:53.602988 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.602750    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-tls-certs\") pod \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") "
Sep 16 08:01:53.602988 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.602786    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-tmp-dir\") pod \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") "
Sep 16 08:01:53.602988 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.602805    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-model-cache\") pod \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") "
Sep 16 08:01:53.602988 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.602912    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-home\") pod \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\" (UID: \"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2\") "
Sep 16 08:01:53.603211 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.603099    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2" (UID: "c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:01:53.603211 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.603165    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-home" (OuterVolumeSpecName: "home") pod "c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2" (UID: "c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:01:53.603211 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.603179    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-model-cache" (OuterVolumeSpecName: "model-cache") pod "c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2" (UID: "c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:01:53.605110 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.605081    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-dshm" (OuterVolumeSpecName: "dshm") pod "c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2" (UID: "c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:01:53.605247 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.605187    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2" (UID: "c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 08:01:53.605303 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.605236    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-kube-api-access-z24md" (OuterVolumeSpecName: "kube-api-access-z24md") pod "c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2" (UID: "c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2"). InnerVolumeSpecName "kube-api-access-z24md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:01:53.648297 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.648260    2581 generic.go:356] "Generic (PLEG): container finished" podID="c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2" containerID="287b7c2f77175dffd8b7aa16a5ab7713040d115c4ec0194019f4bc9d9a727625" exitCode=0
Sep 16 08:01:53.648494 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.648305    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"
Sep 16 08:01:53.648494 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.648340    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf" event={"ID":"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2","Type":"ContainerDied","Data":"287b7c2f77175dffd8b7aa16a5ab7713040d115c4ec0194019f4bc9d9a727625"}
Sep 16 08:01:53.648494 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.648386    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf" event={"ID":"c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2","Type":"ContainerDied","Data":"1515516abb6a2c282f489c89bb5c5a4408338e64ba383a43f538a6e447dcc97d"}
Sep 16 08:01:53.648494 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.648405    2581 scope.go:117] "RemoveContainer" containerID="287b7c2f77175dffd8b7aa16a5ab7713040d115c4ec0194019f4bc9d9a727625"
Sep 16 08:01:53.657685 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.657666    2581 scope.go:117] "RemoveContainer" containerID="287b7c2f77175dffd8b7aa16a5ab7713040d115c4ec0194019f4bc9d9a727625"
Sep 16 08:01:53.657982 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:01:53.657963    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287b7c2f77175dffd8b7aa16a5ab7713040d115c4ec0194019f4bc9d9a727625\": container with ID starting with 287b7c2f77175dffd8b7aa16a5ab7713040d115c4ec0194019f4bc9d9a727625 not found: ID does not exist" containerID="287b7c2f77175dffd8b7aa16a5ab7713040d115c4ec0194019f4bc9d9a727625"
Sep 16 08:01:53.658030 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.657993    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287b7c2f77175dffd8b7aa16a5ab7713040d115c4ec0194019f4bc9d9a727625"} err="failed to get container status \"287b7c2f77175dffd8b7aa16a5ab7713040d115c4ec0194019f4bc9d9a727625\": rpc error: code = NotFound desc = could not find container \"287b7c2f77175dffd8b7aa16a5ab7713040d115c4ec0194019f4bc9d9a727625\": container with ID starting with 287b7c2f77175dffd8b7aa16a5ab7713040d115c4ec0194019f4bc9d9a727625 not found: ID does not exist"
Sep 16 08:01:53.669927 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.669904    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"]
Sep 16 08:01:53.673312 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.673290    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/auth-test-model-e9fc8c-kserve-858f5cd9b7-wswdf"]
Sep 16 08:01:53.683565 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.683542    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="328dfdf4-6723-4d54-8902-2d217f6be6a4" path="/var/lib/kubelet/pods/328dfdf4-6723-4d54-8902-2d217f6be6a4/volumes"
Sep 16 08:01:53.683954 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.683940    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2" path="/var/lib/kubelet/pods/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2/volumes"
Sep 16 08:01:53.703747 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.703693    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:53.703747 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.703713    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z24md\" (UniqueName: \"kubernetes.io/projected/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-kube-api-access-z24md\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:53.703747 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.703722    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:53.703747 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.703732    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:53.703747 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.703741    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:01:53.703747 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:01:53.703749    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:02:24.372707 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.372629    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"]
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373019    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8999f450-ddca-4885-af15-c203abdf47d7" containerName="istio-proxy"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373032    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8999f450-ddca-4885-af15-c203abdf47d7" containerName="istio-proxy"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373044    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7888826b-cd84-4cc6-b808-e18a75041202" containerName="istio-proxy"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373049    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7888826b-cd84-4cc6-b808-e18a75041202" containerName="istio-proxy"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373061    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7888826b-cd84-4cc6-b808-e18a75041202" containerName="istio-proxy"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373069    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7888826b-cd84-4cc6-b808-e18a75041202" containerName="istio-proxy"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373079    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d391f1c7-023e-485b-8709-7f0e4336eafc" containerName="istio-proxy"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373084    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d391f1c7-023e-485b-8709-7f0e4336eafc" containerName="istio-proxy"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373090    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d391f1c7-023e-485b-8709-7f0e4336eafc" containerName="istio-proxy"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373095    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d391f1c7-023e-485b-8709-7f0e4336eafc" containerName="istio-proxy"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373102    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e" containerName="payload-pre-processing"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373107    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e" containerName="payload-pre-processing"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373113    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3418d981-b15f-4ea2-a807-75b731ea0804" containerName="istio-proxy"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373119    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3418d981-b15f-4ea2-a807-75b731ea0804" containerName="istio-proxy"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373124    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="881465a0-eb62-4314-b5d2-eb180a5d600d" containerName="payload-processing"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373130    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="881465a0-eb62-4314-b5d2-eb180a5d600d" containerName="payload-processing"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373144    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41a6be09-4aaf-42c7-a00c-33cb8bcb3f45" containerName="payload-pre-processing"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373150    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41a6be09-4aaf-42c7-a00c-33cb8bcb3f45" containerName="payload-pre-processing"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373157    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06b891c9-f64a-4833-8bd0-df3d3482f956" containerName="main"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373163    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="06b891c9-f64a-4833-8bd0-df3d3482f956" containerName="main"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373173    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2" containerName="main"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373178    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2" containerName="main"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373185    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328dfdf4-6723-4d54-8902-2d217f6be6a4" containerName="main"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373191    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="328dfdf4-6723-4d54-8902-2d217f6be6a4" containerName="main"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373198    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3418d981-b15f-4ea2-a807-75b731ea0804" containerName="istio-proxy"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373203    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3418d981-b15f-4ea2-a807-75b731ea0804" containerName="istio-proxy"
Sep 16 08:02:24.373202 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373211    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a6933b-e889-4ae6-83d8-d512efc75536" containerName="revoke-keys"
Sep 16 08:02:24.374134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373221    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a6933b-e889-4ae6-83d8-d512efc75536" containerName="revoke-keys"
Sep 16 08:02:24.374134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373280    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d391f1c7-023e-485b-8709-7f0e4336eafc" containerName="istio-proxy"
Sep 16 08:02:24.374134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373286    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7888826b-cd84-4cc6-b808-e18a75041202" containerName="istio-proxy"
Sep 16 08:02:24.374134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373295    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8999f450-ddca-4885-af15-c203abdf47d7" containerName="istio-proxy"
Sep 16 08:02:24.374134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373303    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3418d981-b15f-4ea2-a807-75b731ea0804" containerName="istio-proxy"
Sep 16 08:02:24.374134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373309    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="41a6be09-4aaf-42c7-a00c-33cb8bcb3f45" containerName="payload-pre-processing"
Sep 16 08:02:24.374134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373314    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a6933b-e889-4ae6-83d8-d512efc75536" containerName="revoke-keys"
Sep 16 08:02:24.374134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373320    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="79bfb54d-8e46-4fd5-b881-a4ec45ebfd7e" containerName="payload-pre-processing"
Sep 16 08:02:24.374134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373324    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42595f4-91f8-45ff-9a0e-3ca5b75f6ae2" containerName="main"
Sep 16 08:02:24.374134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373330    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="881465a0-eb62-4314-b5d2-eb180a5d600d" containerName="payload-processing"
Sep 16 08:02:24.374134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373334    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7888826b-cd84-4cc6-b808-e18a75041202" containerName="istio-proxy"
Sep 16 08:02:24.374134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373340    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8999f450-ddca-4885-af15-c203abdf47d7" containerName="istio-proxy"
Sep 16 08:02:24.374134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373344    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d391f1c7-023e-485b-8709-7f0e4336eafc" containerName="istio-proxy"
Sep 16 08:02:24.374134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373349    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="328dfdf4-6723-4d54-8902-2d217f6be6a4" containerName="main"
Sep 16 08:02:24.374134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.373355    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="06b891c9-f64a-4833-8bd0-df3d3482f956" containerName="main"
Sep 16 08:02:24.376425 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.376409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.378951 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.378929    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e9fc8c\"/\"openshift-service-ca.crt\""
Sep 16 08:02:24.379997 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.379963    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e9fc8c\"/\"kube-root-ca.crt\""
Sep 16 08:02:24.380108 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.380020    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e9fc8c\"/\"sub-test-model-e9fc8c-kserve-self-signed-certs\""
Sep 16 08:02:24.380108 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.380049    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e9fc8c\"/\"default-dockercfg-bd6xl\""
Sep 16 08:02:24.387909 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.387875    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"]
Sep 16 08:02:24.481684 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.481611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-tmp-dir\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.481910 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.481690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ltfj\" (UniqueName: \"kubernetes.io/projected/6913bb86-4376-4f57-8b3f-3165fd911666-kube-api-access-9ltfj\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.481910 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.481739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-model-cache\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.481910 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.481768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6913bb86-4376-4f57-8b3f-3165fd911666-tls-certs\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.481910 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.481814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-home\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.481910 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.481851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-dshm\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.583122 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.583078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-home\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.583122 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.583121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-dshm\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.583380 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.583177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-tmp-dir\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.583380 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.583212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ltfj\" (UniqueName: \"kubernetes.io/projected/6913bb86-4376-4f57-8b3f-3165fd911666-kube-api-access-9ltfj\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.583380 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.583255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-model-cache\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.583380 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.583286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6913bb86-4376-4f57-8b3f-3165fd911666-tls-certs\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.583578 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.583510    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-home\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.583640 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.583599    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-tmp-dir\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.583701 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.583678    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-model-cache\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.585443 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.585409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-dshm\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.585728 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.585709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6913bb86-4376-4f57-8b3f-3165fd911666-tls-certs\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.591204 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.591179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ltfj\" (UniqueName: \"kubernetes.io/projected/6913bb86-4376-4f57-8b3f-3165fd911666-kube-api-access-9ltfj\") pod \"sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.687632 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.687589    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:24.959659 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:24.959630    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"]
Sep 16 08:02:24.962459 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:02:24.962427    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6913bb86_4376_4f57_8b3f_3165fd911666.slice/crio-d9c2b4ea51269036f380497ce7bf37c856202189269e03801150930a8e221780 WatchSource:0}: Error finding container d9c2b4ea51269036f380497ce7bf37c856202189269e03801150930a8e221780: Status 404 returned error can't find the container with id d9c2b4ea51269036f380497ce7bf37c856202189269e03801150930a8e221780
Sep 16 08:02:25.817727 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:25.817694    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj" event={"ID":"6913bb86-4376-4f57-8b3f-3165fd911666","Type":"ContainerStarted","Data":"21033c9c58c2ccb5c9d9a1aa161822a0ddaf7e0833e7823ccedf566ebe5a6efd"}
Sep 16 08:02:25.818217 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:25.817736    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj" event={"ID":"6913bb86-4376-4f57-8b3f-3165fd911666","Type":"ContainerStarted","Data":"d9c2b4ea51269036f380497ce7bf37c856202189269e03801150930a8e221780"}
Sep 16 08:02:25.837751 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:25.837690    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj" podStartSLOduration=1.837671405 podStartE2EDuration="1.837671405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:02:25.835553829 +0000 UTC m=+1706.762165983" watchObservedRunningTime="2026-09-16 08:02:25.837671405 +0000 UTC m=+1706.764283604"
Sep 16 08:02:34.688546 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:34.688510    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:34.689028 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:34.688560    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:34.700935 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:34.700908    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:02:34.873107 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:02:34.873072    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:03:04.260293 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:04.260259    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"]
Sep 16 08:03:19.279101 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.279044    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj" podUID="6913bb86-4376-4f57-8b3f-3165fd911666" containerName="main" containerID="cri-o://21033c9c58c2ccb5c9d9a1aa161822a0ddaf7e0833e7823ccedf566ebe5a6efd" gracePeriod=45
Sep 16 08:03:19.518961 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.518937    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:03:19.702436 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.702404    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-dshm\") pod \"6913bb86-4376-4f57-8b3f-3165fd911666\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") "
Sep 16 08:03:19.702596 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.702446    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-tmp-dir\") pod \"6913bb86-4376-4f57-8b3f-3165fd911666\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") "
Sep 16 08:03:19.702596 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.702476    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ltfj\" (UniqueName: \"kubernetes.io/projected/6913bb86-4376-4f57-8b3f-3165fd911666-kube-api-access-9ltfj\") pod \"6913bb86-4376-4f57-8b3f-3165fd911666\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") "
Sep 16 08:03:19.702596 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.702565    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-home\") pod \"6913bb86-4376-4f57-8b3f-3165fd911666\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") "
Sep 16 08:03:19.702754 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.702602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6913bb86-4376-4f57-8b3f-3165fd911666-tls-certs\") pod \"6913bb86-4376-4f57-8b3f-3165fd911666\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") "
Sep 16 08:03:19.702754 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.702652    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-model-cache\") pod \"6913bb86-4376-4f57-8b3f-3165fd911666\" (UID: \"6913bb86-4376-4f57-8b3f-3165fd911666\") "
Sep 16 08:03:19.702754 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.702714    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6913bb86-4376-4f57-8b3f-3165fd911666" (UID: "6913bb86-4376-4f57-8b3f-3165fd911666"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:03:19.702956 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.702870    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-home" (OuterVolumeSpecName: "home") pod "6913bb86-4376-4f57-8b3f-3165fd911666" (UID: "6913bb86-4376-4f57-8b3f-3165fd911666"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:03:19.703008 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.702964    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-model-cache" (OuterVolumeSpecName: "model-cache") pod "6913bb86-4376-4f57-8b3f-3165fd911666" (UID: "6913bb86-4376-4f57-8b3f-3165fd911666"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:03:19.703054 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.703041    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:03:19.703100 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.703058    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:03:19.703100 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.703070    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:03:19.704575 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.704553    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-dshm" (OuterVolumeSpecName: "dshm") pod "6913bb86-4376-4f57-8b3f-3165fd911666" (UID: "6913bb86-4376-4f57-8b3f-3165fd911666"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:03:19.704662 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.704652    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6913bb86-4376-4f57-8b3f-3165fd911666-kube-api-access-9ltfj" (OuterVolumeSpecName: "kube-api-access-9ltfj") pod "6913bb86-4376-4f57-8b3f-3165fd911666" (UID: "6913bb86-4376-4f57-8b3f-3165fd911666"). InnerVolumeSpecName "kube-api-access-9ltfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:03:19.704716 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.704698    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6913bb86-4376-4f57-8b3f-3165fd911666-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6913bb86-4376-4f57-8b3f-3165fd911666" (UID: "6913bb86-4376-4f57-8b3f-3165fd911666"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 08:03:19.804307 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.804281    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6913bb86-4376-4f57-8b3f-3165fd911666-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:03:19.804307 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.804306    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6913bb86-4376-4f57-8b3f-3165fd911666-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:03:19.804469 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:19.804314    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ltfj\" (UniqueName: \"kubernetes.io/projected/6913bb86-4376-4f57-8b3f-3165fd911666-kube-api-access-9ltfj\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:03:20.085858 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:20.085738    2581 generic.go:356] "Generic (PLEG): container finished" podID="6913bb86-4376-4f57-8b3f-3165fd911666" containerID="21033c9c58c2ccb5c9d9a1aa161822a0ddaf7e0833e7823ccedf566ebe5a6efd" exitCode=0
Sep 16 08:03:20.085858 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:20.085776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj" event={"ID":"6913bb86-4376-4f57-8b3f-3165fd911666","Type":"ContainerDied","Data":"21033c9c58c2ccb5c9d9a1aa161822a0ddaf7e0833e7823ccedf566ebe5a6efd"}
Sep 16 08:03:20.085858 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:20.085792    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"
Sep 16 08:03:20.085858 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:20.085811    2581 scope.go:117] "RemoveContainer" containerID="21033c9c58c2ccb5c9d9a1aa161822a0ddaf7e0833e7823ccedf566ebe5a6efd"
Sep 16 08:03:20.086186 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:20.085802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj" event={"ID":"6913bb86-4376-4f57-8b3f-3165fd911666","Type":"ContainerDied","Data":"d9c2b4ea51269036f380497ce7bf37c856202189269e03801150930a8e221780"}
Sep 16 08:03:20.095297 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:20.095280    2581 scope.go:117] "RemoveContainer" containerID="21033c9c58c2ccb5c9d9a1aa161822a0ddaf7e0833e7823ccedf566ebe5a6efd"
Sep 16 08:03:20.095574 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:03:20.095554    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21033c9c58c2ccb5c9d9a1aa161822a0ddaf7e0833e7823ccedf566ebe5a6efd\": container with ID starting with 21033c9c58c2ccb5c9d9a1aa161822a0ddaf7e0833e7823ccedf566ebe5a6efd not found: ID does not exist" containerID="21033c9c58c2ccb5c9d9a1aa161822a0ddaf7e0833e7823ccedf566ebe5a6efd"
Sep 16 08:03:20.095650 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:20.095580    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21033c9c58c2ccb5c9d9a1aa161822a0ddaf7e0833e7823ccedf566ebe5a6efd"} err="failed to get container status \"21033c9c58c2ccb5c9d9a1aa161822a0ddaf7e0833e7823ccedf566ebe5a6efd\": rpc error: code = NotFound desc = could not find container \"21033c9c58c2ccb5c9d9a1aa161822a0ddaf7e0833e7823ccedf566ebe5a6efd\": container with ID starting with 21033c9c58c2ccb5c9d9a1aa161822a0ddaf7e0833e7823ccedf566ebe5a6efd not found: ID does not exist"
Sep 16 08:03:20.107308 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:20.107284    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"]
Sep 16 08:03:20.110771 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:20.110745    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/sub-test-model-e9fc8c-kserve-648c8b564c-hcbfj"]
Sep 16 08:03:21.683364 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:21.683335    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6913bb86-4376-4f57-8b3f-3165fd911666" path="/var/lib/kubelet/pods/6913bb86-4376-4f57-8b3f-3165fd911666/volumes"
Sep 16 08:03:42.771690 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.771657    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"]
Sep 16 08:03:42.772250 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.772169    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6913bb86-4376-4f57-8b3f-3165fd911666" containerName="main"
Sep 16 08:03:42.772250 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.772196    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6913bb86-4376-4f57-8b3f-3165fd911666" containerName="main"
Sep 16 08:03:42.772250 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.772225    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8999f450-ddca-4885-af15-c203abdf47d7" containerName="istio-proxy"
Sep 16 08:03:42.772250 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.772235    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8999f450-ddca-4885-af15-c203abdf47d7" containerName="istio-proxy"
Sep 16 08:03:42.772457 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.772330    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6913bb86-4376-4f57-8b3f-3165fd911666" containerName="main"
Sep 16 08:03:42.772457 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.772350    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3418d981-b15f-4ea2-a807-75b731ea0804" containerName="istio-proxy"
Sep 16 08:03:42.775450 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.775431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.778737 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.778712    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e9fc8c\"/\"openshift-service-ca.crt\""
Sep 16 08:03:42.778737 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.778726    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e9fc8c\"/\"kube-root-ca.crt\""
Sep 16 08:03:42.778934 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.778719    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e9fc8c\"/\"default-dockercfg-bd6xl\""
Sep 16 08:03:42.778934 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.778753    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e9fc8c\"/\"rate-test-model-e9fc8c-kserve-self-signed-certs\""
Sep 16 08:03:42.786226 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.786202    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"]
Sep 16 08:03:42.889178 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.889150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-tmp-dir\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.889315 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.889192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-home\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.889315 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.889214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-model-cache\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.889315 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.889241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37d02776-9a50-456d-902b-d72d9292e321-tls-certs\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.889315 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.889288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-dshm\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.889460 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.889328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmjgh\" (UniqueName: \"kubernetes.io/projected/37d02776-9a50-456d-902b-d72d9292e321-kube-api-access-jmjgh\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.990050 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.990015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-tmp-dir\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.990225 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.990069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-home\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.990225 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.990100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-model-cache\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.990225 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.990126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37d02776-9a50-456d-902b-d72d9292e321-tls-certs\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.990225 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.990199    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-dshm\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.990505 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.990286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmjgh\" (UniqueName: \"kubernetes.io/projected/37d02776-9a50-456d-902b-d72d9292e321-kube-api-access-jmjgh\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.990570 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.990500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-tmp-dir\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.990570 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.990517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-model-cache\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.990570 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.990553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-home\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.992399 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.992376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-dshm\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.992609 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.992590    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37d02776-9a50-456d-902b-d72d9292e321-tls-certs\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:42.998525 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:42.998503    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmjgh\" (UniqueName: \"kubernetes.io/projected/37d02776-9a50-456d-902b-d72d9292e321-kube-api-access-jmjgh\") pod \"rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") " pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:43.085617 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:43.085557    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:43.357076 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:43.357050    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"]
Sep 16 08:03:43.358897 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:03:43.358868    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37d02776_9a50_456d_902b_d72d9292e321.slice/crio-0e07214178989ab34ee6f508552a6dfccf287f7d64c6eeae3d5422bddc7b2e26 WatchSource:0}: Error finding container 0e07214178989ab34ee6f508552a6dfccf287f7d64c6eeae3d5422bddc7b2e26: Status 404 returned error can't find the container with id 0e07214178989ab34ee6f508552a6dfccf287f7d64c6eeae3d5422bddc7b2e26
Sep 16 08:03:44.203711 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:44.203670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf" event={"ID":"37d02776-9a50-456d-902b-d72d9292e321","Type":"ContainerStarted","Data":"b0bf585ba73632232972e7b8b376052a8de55138d497cfc2a834d9b908ff1c6e"}
Sep 16 08:03:44.203711 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:44.203714    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf" event={"ID":"37d02776-9a50-456d-902b-d72d9292e321","Type":"ContainerStarted","Data":"0e07214178989ab34ee6f508552a6dfccf287f7d64c6eeae3d5422bddc7b2e26"}
Sep 16 08:03:44.221195 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:44.221142    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf" podStartSLOduration=2.221127887 podStartE2EDuration="2.221127887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:03:44.21995517 +0000 UTC m=+1785.146567324" watchObservedRunningTime="2026-09-16 08:03:44.221127887 +0000 UTC m=+1785.147740040"
Sep 16 08:03:53.086505 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:53.086427    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:53.086505 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:53.086499    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:53.099229 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:53.099207    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:53.258498 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:53.258468    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:03:59.662910 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:59.662875    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 08:03:59.666718 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:03:59.666695    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 08:04:23.652242 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:23.652201    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"]
Sep 16 08:04:24.446087 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.446039    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"]
Sep 16 08:04:24.450134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.450111    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.452718 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.452698    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-796e11-openshift-default-dockercfg-lt4gb\""
Sep 16 08:04:24.460008 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.459971    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"]
Sep 16 08:04:24.547527 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.547492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-token\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.547718 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.547554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxfzn\" (UniqueName: \"kubernetes.io/projected/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-kube-api-access-lxfzn\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.547718 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.547587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-podinfo\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.547718 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.547654    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-workload-certs\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.547718 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.547684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-data\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.547718 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.547704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istiod-ca-cert\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.548029 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.547732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-credential-socket\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.548029 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.547791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-envoy\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.548029 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.547872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-workload-socket\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.649261 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.649225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxfzn\" (UniqueName: \"kubernetes.io/projected/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-kube-api-access-lxfzn\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.649261 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.649267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-podinfo\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.649499 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.649299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-workload-certs\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.649499 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.649317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-data\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.649499 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.649336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istiod-ca-cert\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.649499 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.649353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-credential-socket\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.649499 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.649376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-envoy\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.649758 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.649497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-workload-socket\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.649758 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.649570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-token\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.649885 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.649837    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-data\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.649885 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.649864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-workload-certs\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.650074 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.650053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-credential-socket\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.650142 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.650102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-workload-socket\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.650264 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.650240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istiod-ca-cert\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.651894 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.651869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-podinfo\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.651992 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.651921    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-envoy\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.657057 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.657038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxfzn\" (UniqueName: \"kubernetes.io/projected/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-kube-api-access-lxfzn\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.657398 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.657066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-token\") pod \"e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") " pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:24.763859 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:24.763764    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:25.044080 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:25.044048    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"]
Sep 16 08:04:25.044617 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:04:25.044586    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3fd96c9_b483_44a9_9530_65f20cc8bfe3.slice/crio-7178ab65d53ec2d3a8a14ea50ddccc9458aa4e02cfbb808f8805c39e2c8e8951 WatchSource:0}: Error finding container 7178ab65d53ec2d3a8a14ea50ddccc9458aa4e02cfbb808f8805c39e2c8e8951: Status 404 returned error can't find the container with id 7178ab65d53ec2d3a8a14ea50ddccc9458aa4e02cfbb808f8805c39e2c8e8951
Sep 16 08:04:25.046921 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:25.046885    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 08:04:25.047021 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:25.046954    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 08:04:25.047021 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:25.046989    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 08:04:25.406715 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:25.406681    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx" event={"ID":"f3fd96c9-b483-44a9-9530-65f20cc8bfe3","Type":"ContainerStarted","Data":"24c1ca9d0679669bc7521e7aacb969efddf83c6686f1bd371cb1469476a429d6"}
Sep 16 08:04:25.406890 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:25.406742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx" event={"ID":"f3fd96c9-b483-44a9-9530-65f20cc8bfe3","Type":"ContainerStarted","Data":"7178ab65d53ec2d3a8a14ea50ddccc9458aa4e02cfbb808f8805c39e2c8e8951"}
Sep 16 08:04:25.427697 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:25.427646    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx" podStartSLOduration=1.427627507 podStartE2EDuration="1.427627507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:04:25.425779649 +0000 UTC m=+1826.352391839" watchObservedRunningTime="2026-09-16 08:04:25.427627507 +0000 UTC m=+1826.354239660"
Sep 16 08:04:25.764567 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:25.764492    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:26.769451 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:26.769422    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:27.417224 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:27.417186    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:27.418178 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:27.418159    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:29.917720 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:29.917689    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"]
Sep 16 08:04:29.921140 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:29.921123    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"
Sep 16 08:04:29.923401 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:29.923381    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-796e11\""
Sep 16 08:04:29.923401 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:29.923381    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-796e11-dockercfg-gcbfx\""
Sep 16 08:04:29.931011 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:29.930990    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"]
Sep 16 08:04:29.951358 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:29.951335    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"]
Sep 16 08:04:29.954551 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:29.954533    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"
Sep 16 08:04:29.965751 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:29.965729    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"]
Sep 16 08:04:30.005595 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.005569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj5fx\" (UniqueName: \"kubernetes.io/projected/fc41361f-cd2b-45b1-a90f-8e3858676592-kube-api-access-wj5fx\") pod \"payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn\" (UID: \"fc41361f-cd2b-45b1-a90f-8e3858676592\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"
Sep 16 08:04:30.005709 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.005608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f16370a-811e-4d4c-97d8-2be259b73b5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p\" (UID: \"1f16370a-811e-4d4c-97d8-2be259b73b5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"
Sep 16 08:04:30.005780 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.005712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc41361f-cd2b-45b1-a90f-8e3858676592-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn\" (UID: \"fc41361f-cd2b-45b1-a90f-8e3858676592\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"
Sep 16 08:04:30.005780 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.005765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mmlv\" (UniqueName: \"kubernetes.io/projected/1f16370a-811e-4d4c-97d8-2be259b73b5a-kube-api-access-2mmlv\") pod \"payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p\" (UID: \"1f16370a-811e-4d4c-97d8-2be259b73b5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"
Sep 16 08:04:30.106397 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.106361    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f16370a-811e-4d4c-97d8-2be259b73b5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p\" (UID: \"1f16370a-811e-4d4c-97d8-2be259b73b5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"
Sep 16 08:04:30.106564 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.106443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc41361f-cd2b-45b1-a90f-8e3858676592-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn\" (UID: \"fc41361f-cd2b-45b1-a90f-8e3858676592\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"
Sep 16 08:04:30.106564 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.106485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mmlv\" (UniqueName: \"kubernetes.io/projected/1f16370a-811e-4d4c-97d8-2be259b73b5a-kube-api-access-2mmlv\") pod \"payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p\" (UID: \"1f16370a-811e-4d4c-97d8-2be259b73b5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"
Sep 16 08:04:30.106645 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.106559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj5fx\" (UniqueName: \"kubernetes.io/projected/fc41361f-cd2b-45b1-a90f-8e3858676592-kube-api-access-wj5fx\") pod \"payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn\" (UID: \"fc41361f-cd2b-45b1-a90f-8e3858676592\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"
Sep 16 08:04:30.107163 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.107143    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f16370a-811e-4d4c-97d8-2be259b73b5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p\" (UID: \"1f16370a-811e-4d4c-97d8-2be259b73b5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"
Sep 16 08:04:30.107216 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.107148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc41361f-cd2b-45b1-a90f-8e3858676592-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn\" (UID: \"fc41361f-cd2b-45b1-a90f-8e3858676592\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"
Sep 16 08:04:30.114629 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.114598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj5fx\" (UniqueName: \"kubernetes.io/projected/fc41361f-cd2b-45b1-a90f-8e3858676592-kube-api-access-wj5fx\") pod \"payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn\" (UID: \"fc41361f-cd2b-45b1-a90f-8e3858676592\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"
Sep 16 08:04:30.114629 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.114618    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mmlv\" (UniqueName: \"kubernetes.io/projected/1f16370a-811e-4d4c-97d8-2be259b73b5a-kube-api-access-2mmlv\") pod \"payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p\" (UID: \"1f16370a-811e-4d4c-97d8-2be259b73b5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"
Sep 16 08:04:30.232973 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.232903    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"
Sep 16 08:04:30.264528 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.264501    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"
Sep 16 08:04:30.503005 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.502978    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"]
Sep 16 08:04:30.505079 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:04:30.505039    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc41361f_cd2b_45b1_a90f_8e3858676592.slice/crio-26a0fd6ccceaf49accab7d659e43bbc97d251e37ac0739088df94148f7c64787 WatchSource:0}: Error finding container 26a0fd6ccceaf49accab7d659e43bbc97d251e37ac0739088df94148f7c64787: Status 404 returned error can't find the container with id 26a0fd6ccceaf49accab7d659e43bbc97d251e37ac0739088df94148f7c64787
Sep 16 08:04:30.545290 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:30.545263    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"]
Sep 16 08:04:30.547152 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:04:30.547132    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f16370a_811e_4d4c_97d8_2be259b73b5a.slice/crio-60d6c724580897bc306f45ec24239fe560f3df9ef8699088484691f30dfb5f95 WatchSource:0}: Error finding container 60d6c724580897bc306f45ec24239fe560f3df9ef8699088484691f30dfb5f95: Status 404 returned error can't find the container with id 60d6c724580897bc306f45ec24239fe560f3df9ef8699088484691f30dfb5f95
Sep 16 08:04:31.438221 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:31.438181    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn" event={"ID":"fc41361f-cd2b-45b1-a90f-8e3858676592","Type":"ContainerStarted","Data":"442be6aa205886a8ac253fda3f3ac4adebb310af6c26600b9c3449a839bbbbbe"}
Sep 16 08:04:31.438682 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:31.438411    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"
Sep 16 08:04:31.438682 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:31.438464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn" event={"ID":"fc41361f-cd2b-45b1-a90f-8e3858676592","Type":"ContainerStarted","Data":"26a0fd6ccceaf49accab7d659e43bbc97d251e37ac0739088df94148f7c64787"}
Sep 16 08:04:31.440309 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:31.440282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p" event={"ID":"1f16370a-811e-4d4c-97d8-2be259b73b5a","Type":"ContainerStarted","Data":"85d066aa1d3f0d3e5cfae4b8183daade58a54601a3df5ff63c968adef13b5ab3"}
Sep 16 08:04:31.440412 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:31.440330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p" event={"ID":"1f16370a-811e-4d4c-97d8-2be259b73b5a","Type":"ContainerStarted","Data":"60d6c724580897bc306f45ec24239fe560f3df9ef8699088484691f30dfb5f95"}
Sep 16 08:04:31.440412 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:31.440350    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"
Sep 16 08:04:31.454443 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:31.454390    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn" podStartSLOduration=2.454375729 podStartE2EDuration="2.454375729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:04:31.453511464 +0000 UTC m=+1832.380123627" watchObservedRunningTime="2026-09-16 08:04:31.454375729 +0000 UTC m=+1832.380987880"
Sep 16 08:04:31.469263 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:31.469225    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p" podStartSLOduration=2.4692140399999998 podStartE2EDuration="2.46921404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:04:31.467674346 +0000 UTC m=+1832.394286495" watchObservedRunningTime="2026-09-16 08:04:31.46921404 +0000 UTC m=+1832.395826192"
Sep 16 08:04:38.671149 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:38.671108    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf" podUID="37d02776-9a50-456d-902b-d72d9292e321" containerName="main" containerID="cri-o://b0bf585ba73632232972e7b8b376052a8de55138d497cfc2a834d9b908ff1c6e" gracePeriod=45
Sep 16 08:04:38.959748 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:38.959724    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:04:39.091356 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.091318    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-dshm\") pod \"37d02776-9a50-456d-902b-d72d9292e321\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") "
Sep 16 08:04:39.091532 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.091378    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-tmp-dir\") pod \"37d02776-9a50-456d-902b-d72d9292e321\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") "
Sep 16 08:04:39.091532 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.091405    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-model-cache\") pod \"37d02776-9a50-456d-902b-d72d9292e321\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") "
Sep 16 08:04:39.091532 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.091446    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-home\") pod \"37d02776-9a50-456d-902b-d72d9292e321\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") "
Sep 16 08:04:39.091665 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.091557    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37d02776-9a50-456d-902b-d72d9292e321-tls-certs\") pod \"37d02776-9a50-456d-902b-d72d9292e321\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") "
Sep 16 08:04:39.091665 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.091601    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmjgh\" (UniqueName: \"kubernetes.io/projected/37d02776-9a50-456d-902b-d72d9292e321-kube-api-access-jmjgh\") pod \"37d02776-9a50-456d-902b-d72d9292e321\" (UID: \"37d02776-9a50-456d-902b-d72d9292e321\") "
Sep 16 08:04:39.091779 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.091669    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37d02776-9a50-456d-902b-d72d9292e321" (UID: "37d02776-9a50-456d-902b-d72d9292e321"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:04:39.091779 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.091679    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-model-cache" (OuterVolumeSpecName: "model-cache") pod "37d02776-9a50-456d-902b-d72d9292e321" (UID: "37d02776-9a50-456d-902b-d72d9292e321"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:04:39.091779 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.091735    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-home" (OuterVolumeSpecName: "home") pod "37d02776-9a50-456d-902b-d72d9292e321" (UID: "37d02776-9a50-456d-902b-d72d9292e321"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:04:39.091977 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.091922    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:04:39.091977 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.091940    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:04:39.091977 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.091952    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:04:39.093466 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.093442    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-dshm" (OuterVolumeSpecName: "dshm") pod "37d02776-9a50-456d-902b-d72d9292e321" (UID: "37d02776-9a50-456d-902b-d72d9292e321"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:04:39.093733 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.093706    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d02776-9a50-456d-902b-d72d9292e321-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37d02776-9a50-456d-902b-d72d9292e321" (UID: "37d02776-9a50-456d-902b-d72d9292e321"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 08:04:39.093812 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.093731    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d02776-9a50-456d-902b-d72d9292e321-kube-api-access-jmjgh" (OuterVolumeSpecName: "kube-api-access-jmjgh") pod "37d02776-9a50-456d-902b-d72d9292e321" (UID: "37d02776-9a50-456d-902b-d72d9292e321"). InnerVolumeSpecName "kube-api-access-jmjgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:04:39.193383 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.193298    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37d02776-9a50-456d-902b-d72d9292e321-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:04:39.193383 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.193328    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmjgh\" (UniqueName: \"kubernetes.io/projected/37d02776-9a50-456d-902b-d72d9292e321-kube-api-access-jmjgh\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:04:39.193383 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.193337    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37d02776-9a50-456d-902b-d72d9292e321-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:04:39.482579 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.482497    2581 generic.go:356] "Generic (PLEG): container finished" podID="37d02776-9a50-456d-902b-d72d9292e321" containerID="b0bf585ba73632232972e7b8b376052a8de55138d497cfc2a834d9b908ff1c6e" exitCode=0
Sep 16 08:04:39.482579 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.482537    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"
Sep 16 08:04:39.482763 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.482580    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf" event={"ID":"37d02776-9a50-456d-902b-d72d9292e321","Type":"ContainerDied","Data":"b0bf585ba73632232972e7b8b376052a8de55138d497cfc2a834d9b908ff1c6e"}
Sep 16 08:04:39.482763 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.482623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf" event={"ID":"37d02776-9a50-456d-902b-d72d9292e321","Type":"ContainerDied","Data":"0e07214178989ab34ee6f508552a6dfccf287f7d64c6eeae3d5422bddc7b2e26"}
Sep 16 08:04:39.482763 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.482641    2581 scope.go:117] "RemoveContainer" containerID="b0bf585ba73632232972e7b8b376052a8de55138d497cfc2a834d9b908ff1c6e"
Sep 16 08:04:39.494403 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.494382    2581 scope.go:117] "RemoveContainer" containerID="b0bf585ba73632232972e7b8b376052a8de55138d497cfc2a834d9b908ff1c6e"
Sep 16 08:04:39.494695 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:04:39.494669    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0bf585ba73632232972e7b8b376052a8de55138d497cfc2a834d9b908ff1c6e\": container with ID starting with b0bf585ba73632232972e7b8b376052a8de55138d497cfc2a834d9b908ff1c6e not found: ID does not exist" containerID="b0bf585ba73632232972e7b8b376052a8de55138d497cfc2a834d9b908ff1c6e"
Sep 16 08:04:39.494695 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.494706    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0bf585ba73632232972e7b8b376052a8de55138d497cfc2a834d9b908ff1c6e"} err="failed to get container status \"b0bf585ba73632232972e7b8b376052a8de55138d497cfc2a834d9b908ff1c6e\": rpc error: code = NotFound desc = could not find container \"b0bf585ba73632232972e7b8b376052a8de55138d497cfc2a834d9b908ff1c6e\": container with ID starting with b0bf585ba73632232972e7b8b376052a8de55138d497cfc2a834d9b908ff1c6e not found: ID does not exist"
Sep 16 08:04:39.504644 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.504619    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"]
Sep 16 08:04:39.508676 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.508655    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e9fc8c/rate-test-model-e9fc8c-kserve-6dbc4d5bc5-nhpsf"]
Sep 16 08:04:39.683870 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:39.683836    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37d02776-9a50-456d-902b-d72d9292e321" path="/var/lib/kubelet/pods/37d02776-9a50-456d-902b-d72d9292e321/volumes"
Sep 16 08:04:42.447994 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:42.447962    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"
Sep 16 08:04:42.448418 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:42.448125    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"
Sep 16 08:04:43.458962 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:43.458922    2581 patch_prober.go:28] interesting pod/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 08:04:43.459401 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:43.459010    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx" podUID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 08:04:44.513174 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:44.513141    2581 generic.go:356] "Generic (PLEG): container finished" podID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" containerID="24c1ca9d0679669bc7521e7aacb969efddf83c6686f1bd371cb1469476a429d6" exitCode=0
Sep 16 08:04:44.513572 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:44.513216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx" event={"ID":"f3fd96c9-b483-44a9-9530-65f20cc8bfe3","Type":"ContainerDied","Data":"24c1ca9d0679669bc7521e7aacb969efddf83c6686f1bd371cb1469476a429d6"}
Sep 16 08:04:44.513655 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:44.513639    2581 scope.go:117] "RemoveContainer" containerID="24c1ca9d0679669bc7521e7aacb969efddf83c6686f1bd371cb1469476a429d6"
Sep 16 08:04:44.514854 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:44.514797    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 08:04:44.514934 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:44.514918    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 08:04:44.514973 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:44.514960    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 08:04:45.524834 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:45.524780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx" event={"ID":"f3fd96c9-b483-44a9-9530-65f20cc8bfe3","Type":"ContainerStarted","Data":"679ac30484983e0e21fabfe37f25836f2db7f6d3a507d3b827299cf5e4d0fcc0"}
Sep 16 08:04:45.764767 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:45.764739    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:45.765012 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:45.764989    2581 patch_prober.go:28] interesting pod/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 16 08:04:45.765066 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:45.765036    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx" podUID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 16 08:04:46.764695 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:46.764660    2581 patch_prober.go:28] interesting pod/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 16 08:04:46.765105 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:46.764713    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx" podUID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 16 08:04:47.765066 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:47.765026    2581 patch_prober.go:28] interesting pod/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 16 08:04:47.765467 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:47.765084    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx" podUID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 16 08:04:48.764193 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:48.764155    2581 patch_prober.go:28] interesting pod/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 16 08:04:48.764380 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:48.764224    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx" podUID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 16 08:04:49.764400 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:49.764362    2581 patch_prober.go:28] interesting pod/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 16 08:04:49.764774 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:49.764424    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx" podUID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 16 08:04:50.768421 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:50.768390    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:50.768890 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:50.768650    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:50.769241 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:50.769221    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:04:59.697636 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.697602    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"]
Sep 16 08:04:59.698234 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.698192    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d02776-9a50-456d-902b-d72d9292e321" containerName="main"
Sep 16 08:04:59.698234 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.698212    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d02776-9a50-456d-902b-d72d9292e321" containerName="main"
Sep 16 08:04:59.698361 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.698298    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d02776-9a50-456d-902b-d72d9292e321" containerName="main"
Sep 16 08:04:59.700793 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.700773    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"
Sep 16 08:04:59.703118 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.703100    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5493e8\""
Sep 16 08:04:59.703251 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.703232    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5493e8-dockercfg-7jwxm\""
Sep 16 08:04:59.711962 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.711933    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"]
Sep 16 08:04:59.734633 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.734593    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"]
Sep 16 08:04:59.737097 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.737076    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"
Sep 16 08:04:59.750135 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.750112    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"]
Sep 16 08:04:59.796306 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.795958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47dd702-1136-4b91-b42c-b9153b93fc49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5\" (UID: \"f47dd702-1136-4b91-b42c-b9153b93fc49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"
Sep 16 08:04:59.796306 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.796097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsxnh\" (UniqueName: \"kubernetes.io/projected/460d7a26-c5fe-47b0-8a5e-bcc6e318ac44-kube-api-access-qsxnh\") pod \"payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp\" (UID: \"460d7a26-c5fe-47b0-8a5e-bcc6e318ac44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"
Sep 16 08:04:59.796306 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.796145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460d7a26-c5fe-47b0-8a5e-bcc6e318ac44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp\" (UID: \"460d7a26-c5fe-47b0-8a5e-bcc6e318ac44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"
Sep 16 08:04:59.796306 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.796219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz7hf\" (UniqueName: \"kubernetes.io/projected/f47dd702-1136-4b91-b42c-b9153b93fc49-kube-api-access-xz7hf\") pod \"payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5\" (UID: \"f47dd702-1136-4b91-b42c-b9153b93fc49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"
Sep 16 08:04:59.897260 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.897222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460d7a26-c5fe-47b0-8a5e-bcc6e318ac44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp\" (UID: \"460d7a26-c5fe-47b0-8a5e-bcc6e318ac44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"
Sep 16 08:04:59.897432 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.897304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz7hf\" (UniqueName: \"kubernetes.io/projected/f47dd702-1136-4b91-b42c-b9153b93fc49-kube-api-access-xz7hf\") pod \"payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5\" (UID: \"f47dd702-1136-4b91-b42c-b9153b93fc49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"
Sep 16 08:04:59.897432 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.897339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47dd702-1136-4b91-b42c-b9153b93fc49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5\" (UID: \"f47dd702-1136-4b91-b42c-b9153b93fc49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"
Sep 16 08:04:59.897522 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.897431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsxnh\" (UniqueName: \"kubernetes.io/projected/460d7a26-c5fe-47b0-8a5e-bcc6e318ac44-kube-api-access-qsxnh\") pod \"payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp\" (UID: \"460d7a26-c5fe-47b0-8a5e-bcc6e318ac44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"
Sep 16 08:04:59.898079 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.898062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460d7a26-c5fe-47b0-8a5e-bcc6e318ac44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp\" (UID: \"460d7a26-c5fe-47b0-8a5e-bcc6e318ac44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"
Sep 16 08:04:59.898139 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.898106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47dd702-1136-4b91-b42c-b9153b93fc49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5\" (UID: \"f47dd702-1136-4b91-b42c-b9153b93fc49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"
Sep 16 08:04:59.905179 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.905156    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz7hf\" (UniqueName: \"kubernetes.io/projected/f47dd702-1136-4b91-b42c-b9153b93fc49-kube-api-access-xz7hf\") pod \"payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5\" (UID: \"f47dd702-1136-4b91-b42c-b9153b93fc49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"
Sep 16 08:04:59.905932 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:04:59.905912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsxnh\" (UniqueName: \"kubernetes.io/projected/460d7a26-c5fe-47b0-8a5e-bcc6e318ac44-kube-api-access-qsxnh\") pod \"payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp\" (UID: \"460d7a26-c5fe-47b0-8a5e-bcc6e318ac44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"
Sep 16 08:05:00.012464 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:00.012431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"
Sep 16 08:05:00.046919 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:00.046885    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"
Sep 16 08:05:00.286845 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:00.286799    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"]
Sep 16 08:05:00.288316 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:05:00.288294    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460d7a26_c5fe_47b0_8a5e_bcc6e318ac44.slice/crio-265cac75027050fd992c3c24dcccfb33ebe2e19f742391fb6fdf959962ad5394 WatchSource:0}: Error finding container 265cac75027050fd992c3c24dcccfb33ebe2e19f742391fb6fdf959962ad5394: Status 404 returned error can't find the container with id 265cac75027050fd992c3c24dcccfb33ebe2e19f742391fb6fdf959962ad5394
Sep 16 08:05:00.327546 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:00.327516    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"]
Sep 16 08:05:00.342638 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:05:00.342602    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47dd702_1136_4b91_b42c_b9153b93fc49.slice/crio-964e46ca650ffd5d5196163ce4d2eb74f3815aeb50ef67a9aa21e1821364f320 WatchSource:0}: Error finding container 964e46ca650ffd5d5196163ce4d2eb74f3815aeb50ef67a9aa21e1821364f320: Status 404 returned error can't find the container with id 964e46ca650ffd5d5196163ce4d2eb74f3815aeb50ef67a9aa21e1821364f320
Sep 16 08:05:00.607851 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:00.607733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp" event={"ID":"460d7a26-c5fe-47b0-8a5e-bcc6e318ac44","Type":"ContainerStarted","Data":"378e57ab8e0e1c8cc5f2be3ad1ab2ea0e0bcf35589325ce0c5a7d56d2b3eaa42"}
Sep 16 08:05:00.608104 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:00.608080    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"
Sep 16 08:05:00.608237 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:00.608117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp" event={"ID":"460d7a26-c5fe-47b0-8a5e-bcc6e318ac44","Type":"ContainerStarted","Data":"265cac75027050fd992c3c24dcccfb33ebe2e19f742391fb6fdf959962ad5394"}
Sep 16 08:05:00.609449 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:00.609427    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5" event={"ID":"f47dd702-1136-4b91-b42c-b9153b93fc49","Type":"ContainerStarted","Data":"462ac42ee413df3a7a999bfd805a28348f76289da4b8530d0e19f1efd7260aeb"}
Sep 16 08:05:00.609544 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:00.609463    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5" event={"ID":"f47dd702-1136-4b91-b42c-b9153b93fc49","Type":"ContainerStarted","Data":"964e46ca650ffd5d5196163ce4d2eb74f3815aeb50ef67a9aa21e1821364f320"}
Sep 16 08:05:00.609544 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:00.609503    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"
Sep 16 08:05:00.624246 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:00.624203    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp" podStartSLOduration=1.624188204 podStartE2EDuration="1.624188204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:05:00.623577283 +0000 UTC m=+1861.550189438" watchObservedRunningTime="2026-09-16 08:05:00.624188204 +0000 UTC m=+1861.550800357"
Sep 16 08:05:00.639067 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:00.639019    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5" podStartSLOduration=1.639004726 podStartE2EDuration="1.639004726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:05:00.637615069 +0000 UTC m=+1861.564227223" watchObservedRunningTime="2026-09-16 08:05:00.639004726 +0000 UTC m=+1861.565616878"
Sep 16 08:05:11.617013 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:11.616975    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"
Sep 16 08:05:11.617497 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:11.617262    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"
Sep 16 08:05:27.108208 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:27.108127    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5_f47dd702-1136-4b91-b42c-b9153b93fc49/payload-processing/0.log"
Sep 16 08:05:59.963877 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:05:59.963846    2581 scope.go:117] "RemoveContainer" containerID="39440163c4281d3993c8a3de8e1806f3e1606a96616cb5eb5bda134dccfba721"
Sep 16 08:06:03.010764 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:03.010728    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p_1f16370a-811e-4d4c-97d8-2be259b73b5a/payload-processing/0.log"
Sep 16 08:06:03.247648 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:03.247612    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5_f47dd702-1136-4b91-b42c-b9153b93fc49/payload-processing/0.log"
Sep 16 08:06:03.776872 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:03.776805    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"]
Sep 16 08:06:03.781106 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:03.781074    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:03.783876 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:03.783810    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-d26a85-openshift-default-dockercfg-wppbf\""
Sep 16 08:06:03.793033 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:03.793012    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"]
Sep 16 08:06:03.929542 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:03.929493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-workload-socket\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:03.929542 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:03.929545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-podinfo\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:03.929772 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:03.929644    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-envoy\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:03.929772 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:03.929730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-token\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:03.929772 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:03.929764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204440a6-7b16-4780-8c29-37d3bd4bdd29-istiod-ca-cert\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:03.929958 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:03.929795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-data\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:03.929958 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:03.929878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-credential-socket\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:03.929958 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:03.929927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-workload-certs\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:03.930079 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:03.930021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d7z5\" (UniqueName: \"kubernetes.io/projected/204440a6-7b16-4780-8c29-37d3bd4bdd29-kube-api-access-6d7z5\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.030681 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.030570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-workload-socket\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.030681 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.030623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-podinfo\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.030681 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.030671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-envoy\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.031316 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.030716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-token\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.031316 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.030880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204440a6-7b16-4780-8c29-37d3bd4bdd29-istiod-ca-cert\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.031316 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.030948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-data\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.031316 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.030980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-credential-socket\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.031316 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.031015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-workload-certs\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.031316 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.031063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-workload-socket\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.031316 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.031117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7z5\" (UniqueName: \"kubernetes.io/projected/204440a6-7b16-4780-8c29-37d3bd4bdd29-kube-api-access-6d7z5\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.031316 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.031308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-data\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.031701 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.031569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204440a6-7b16-4780-8c29-37d3bd4bdd29-istiod-ca-cert\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.031701 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.031586    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-credential-socket\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.031701 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.031624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-workload-certs\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.033095 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.033064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-envoy\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.033360 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.033334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-podinfo\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.039075 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.039046    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-token\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.039177 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.039132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d7z5\" (UniqueName: \"kubernetes.io/projected/204440a6-7b16-4780-8c29-37d3bd4bdd29-kube-api-access-6d7z5\") pod \"e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") " pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.093116 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.093074    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:04.372520 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.372494    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"]
Sep 16 08:06:04.374427 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:06:04.374394    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204440a6_7b16_4780_8c29_37d3bd4bdd29.slice/crio-7ded03bc567f000f7f888c9c0aed0f233d21133b2fbe368a936b0e16a5e65a14 WatchSource:0}: Error finding container 7ded03bc567f000f7f888c9c0aed0f233d21133b2fbe368a936b0e16a5e65a14: Status 404 returned error can't find the container with id 7ded03bc567f000f7f888c9c0aed0f233d21133b2fbe368a936b0e16a5e65a14
Sep 16 08:06:04.376793 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.376761    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 08:06:04.376945 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.376884    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 08:06:04.376945 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.376932    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 08:06:04.942077 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.942037    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9" event={"ID":"204440a6-7b16-4780-8c29-37d3bd4bdd29","Type":"ContainerStarted","Data":"313b2944d904ce3e3c224c0473095022645780436b30920dbd79138bdad4724d"}
Sep 16 08:06:04.942265 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.942096    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9" event={"ID":"204440a6-7b16-4780-8c29-37d3bd4bdd29","Type":"ContainerStarted","Data":"7ded03bc567f000f7f888c9c0aed0f233d21133b2fbe368a936b0e16a5e65a14"}
Sep 16 08:06:04.965650 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:04.965597    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9" podStartSLOduration=1.9655803889999999 podStartE2EDuration="1.965580389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:06:04.964372586 +0000 UTC m=+1925.890984754" watchObservedRunningTime="2026-09-16 08:06:04.965580389 +0000 UTC m=+1925.892192542"
Sep 16 08:06:05.093328 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:05.093284    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:05.094923 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:05.094894    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 16 08:06:05.095015 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:05.094951    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9" podUID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 16 08:06:06.098583 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:06.098552    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:06.952476 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:06.952434    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:06.953389 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:06.953368    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:10.447415 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:10.447384    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"]
Sep 16 08:06:10.450929 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:10.450900    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"
Sep 16 08:06:10.453361 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:10.453340    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-d26a85\""
Sep 16 08:06:10.453495 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:10.453417    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-d26a85-dockercfg-z956z\""
Sep 16 08:06:10.459593 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:10.459567    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"]
Sep 16 08:06:10.500068 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:10.500034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e464e515-e843-43b6-a086-10e90ceb6c6d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw\" (UID: \"e464e515-e843-43b6-a086-10e90ceb6c6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"
Sep 16 08:06:10.500251 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:10.500139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b8sq\" (UniqueName: \"kubernetes.io/projected/e464e515-e843-43b6-a086-10e90ceb6c6d-kube-api-access-5b8sq\") pod \"payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw\" (UID: \"e464e515-e843-43b6-a086-10e90ceb6c6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"
Sep 16 08:06:10.600656 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:10.600621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b8sq\" (UniqueName: \"kubernetes.io/projected/e464e515-e843-43b6-a086-10e90ceb6c6d-kube-api-access-5b8sq\") pod \"payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw\" (UID: \"e464e515-e843-43b6-a086-10e90ceb6c6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"
Sep 16 08:06:10.600932 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:10.600741    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e464e515-e843-43b6-a086-10e90ceb6c6d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw\" (UID: \"e464e515-e843-43b6-a086-10e90ceb6c6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"
Sep 16 08:06:10.601458 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:10.601436    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e464e515-e843-43b6-a086-10e90ceb6c6d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw\" (UID: \"e464e515-e843-43b6-a086-10e90ceb6c6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"
Sep 16 08:06:10.609069 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:10.609045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b8sq\" (UniqueName: \"kubernetes.io/projected/e464e515-e843-43b6-a086-10e90ceb6c6d-kube-api-access-5b8sq\") pod \"payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw\" (UID: \"e464e515-e843-43b6-a086-10e90ceb6c6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"
Sep 16 08:06:10.763529 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:10.763438    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"
Sep 16 08:06:11.037908 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:11.037855    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"]
Sep 16 08:06:11.040405 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:06:11.040367    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode464e515_e843_43b6_a086_10e90ceb6c6d.slice/crio-a46008fdaf3c4372ca2e2af70aec2467c9d563c461f362ced3be527192a06aaa WatchSource:0}: Error finding container a46008fdaf3c4372ca2e2af70aec2467c9d563c461f362ced3be527192a06aaa: Status 404 returned error can't find the container with id a46008fdaf3c4372ca2e2af70aec2467c9d563c461f362ced3be527192a06aaa
Sep 16 08:06:11.992391 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:11.992341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw" event={"ID":"e464e515-e843-43b6-a086-10e90ceb6c6d","Type":"ContainerStarted","Data":"275a19c574b50d9a4c19b3828d63d46da44b7cb7ee166abf7879d7385d7c33f3"}
Sep 16 08:06:11.992931 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:11.992461    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw" event={"ID":"e464e515-e843-43b6-a086-10e90ceb6c6d","Type":"ContainerStarted","Data":"a46008fdaf3c4372ca2e2af70aec2467c9d563c461f362ced3be527192a06aaa"}
Sep 16 08:06:11.992931 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:11.992762    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"
Sep 16 08:06:12.011260 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:12.011200    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw" podStartSLOduration=2.01118518 podStartE2EDuration="2.01118518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:06:12.009712605 +0000 UTC m=+1932.936324772" watchObservedRunningTime="2026-09-16 08:06:12.01118518 +0000 UTC m=+1932.937797333"
Sep 16 08:06:23.002378 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:23.002340    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"
Sep 16 08:06:28.088946 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:28.088912    2581 generic.go:356] "Generic (PLEG): container finished" podID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerID="313b2944d904ce3e3c224c0473095022645780436b30920dbd79138bdad4724d" exitCode=0
Sep 16 08:06:28.088946 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:28.088957    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9" event={"ID":"204440a6-7b16-4780-8c29-37d3bd4bdd29","Type":"ContainerDied","Data":"313b2944d904ce3e3c224c0473095022645780436b30920dbd79138bdad4724d"}
Sep 16 08:06:28.089441 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:28.089418    2581 scope.go:117] "RemoveContainer" containerID="313b2944d904ce3e3c224c0473095022645780436b30920dbd79138bdad4724d"
Sep 16 08:06:28.090717 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:28.090682    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 08:06:28.090782 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:28.090763    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 08:06:28.090843 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:28.090799    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 08:06:29.095542 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:29.095508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9" event={"ID":"204440a6-7b16-4780-8c29-37d3bd4bdd29","Type":"ContainerStarted","Data":"c78fbc68cd108c65f87ef7def89140b728e66e4fa7489a119cb6b7f2d6f2da70"}
Sep 16 08:06:30.093250 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:30.093212    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:30.093548 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:30.093522    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 16 08:06:30.093614 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:30.093578    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9" podUID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 16 08:06:31.093637 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:31.093596    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 16 08:06:31.094153 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:31.093661    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9" podUID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 16 08:06:32.093441 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:32.093406    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 16 08:06:32.093630 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:32.093463    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9" podUID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 16 08:06:33.093843 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:33.093782    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 16 08:06:33.094247 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:33.093872    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9" podUID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 16 08:06:34.093880 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:34.093813    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 16 08:06:34.094286 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:34.093903    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9" podUID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 16 08:06:35.098162 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:35.098126    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:35.098623 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:35.098403    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:35.099224 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:35.099201    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:39.534837 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:39.534792    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"]
Sep 16 08:06:39.535312 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:39.535094    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw" podUID="e464e515-e843-43b6-a086-10e90ceb6c6d" containerName="payload-processing" containerID="cri-o://275a19c574b50d9a4c19b3828d63d46da44b7cb7ee166abf7879d7385d7c33f3" gracePeriod=30
Sep 16 08:06:39.787747 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:39.787685    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"
Sep 16 08:06:39.828338 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:39.828307    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e464e515-e843-43b6-a086-10e90ceb6c6d-plugins-config-volume\") pod \"e464e515-e843-43b6-a086-10e90ceb6c6d\" (UID: \"e464e515-e843-43b6-a086-10e90ceb6c6d\") "
Sep 16 08:06:39.828546 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:39.828404    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b8sq\" (UniqueName: \"kubernetes.io/projected/e464e515-e843-43b6-a086-10e90ceb6c6d-kube-api-access-5b8sq\") pod \"e464e515-e843-43b6-a086-10e90ceb6c6d\" (UID: \"e464e515-e843-43b6-a086-10e90ceb6c6d\") "
Sep 16 08:06:39.828715 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:39.828692    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e464e515-e843-43b6-a086-10e90ceb6c6d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e464e515-e843-43b6-a086-10e90ceb6c6d" (UID: "e464e515-e843-43b6-a086-10e90ceb6c6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:06:39.830635 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:39.830606    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e464e515-e843-43b6-a086-10e90ceb6c6d-kube-api-access-5b8sq" (OuterVolumeSpecName: "kube-api-access-5b8sq") pod "e464e515-e843-43b6-a086-10e90ceb6c6d" (UID: "e464e515-e843-43b6-a086-10e90ceb6c6d"). InnerVolumeSpecName "kube-api-access-5b8sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:06:39.929237 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:39.929202    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e464e515-e843-43b6-a086-10e90ceb6c6d-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:06:39.929237 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:39.929231    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b8sq\" (UniqueName: \"kubernetes.io/projected/e464e515-e843-43b6-a086-10e90ceb6c6d-kube-api-access-5b8sq\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:06:40.158536 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:40.158495    2581 generic.go:356] "Generic (PLEG): container finished" podID="e464e515-e843-43b6-a086-10e90ceb6c6d" containerID="275a19c574b50d9a4c19b3828d63d46da44b7cb7ee166abf7879d7385d7c33f3" exitCode=0
Sep 16 08:06:40.158719 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:40.158539    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"
Sep 16 08:06:40.158719 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:40.158584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw" event={"ID":"e464e515-e843-43b6-a086-10e90ceb6c6d","Type":"ContainerDied","Data":"275a19c574b50d9a4c19b3828d63d46da44b7cb7ee166abf7879d7385d7c33f3"}
Sep 16 08:06:40.158719 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:40.158645    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw" event={"ID":"e464e515-e843-43b6-a086-10e90ceb6c6d","Type":"ContainerDied","Data":"a46008fdaf3c4372ca2e2af70aec2467c9d563c461f362ced3be527192a06aaa"}
Sep 16 08:06:40.158719 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:40.158677    2581 scope.go:117] "RemoveContainer" containerID="275a19c574b50d9a4c19b3828d63d46da44b7cb7ee166abf7879d7385d7c33f3"
Sep 16 08:06:40.172593 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:40.171498    2581 scope.go:117] "RemoveContainer" containerID="275a19c574b50d9a4c19b3828d63d46da44b7cb7ee166abf7879d7385d7c33f3"
Sep 16 08:06:40.172593 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:06:40.171930    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"275a19c574b50d9a4c19b3828d63d46da44b7cb7ee166abf7879d7385d7c33f3\": container with ID starting with 275a19c574b50d9a4c19b3828d63d46da44b7cb7ee166abf7879d7385d7c33f3 not found: ID does not exist" containerID="275a19c574b50d9a4c19b3828d63d46da44b7cb7ee166abf7879d7385d7c33f3"
Sep 16 08:06:40.172593 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:40.171972    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"275a19c574b50d9a4c19b3828d63d46da44b7cb7ee166abf7879d7385d7c33f3"} err="failed to get container status \"275a19c574b50d9a4c19b3828d63d46da44b7cb7ee166abf7879d7385d7c33f3\": rpc error: code = NotFound desc = could not find container \"275a19c574b50d9a4c19b3828d63d46da44b7cb7ee166abf7879d7385d7c33f3\": container with ID starting with 275a19c574b50d9a4c19b3828d63d46da44b7cb7ee166abf7879d7385d7c33f3 not found: ID does not exist"
Sep 16 08:06:40.194700 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:40.194541    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"]
Sep 16 08:06:40.196448 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:40.196419    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d26a85-56f95dbb87-bv6gw"]
Sep 16 08:06:41.684148 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:41.684116    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e464e515-e843-43b6-a086-10e90ceb6c6d" path="/var/lib/kubelet/pods/e464e515-e843-43b6-a086-10e90ceb6c6d/volumes"
Sep 16 08:06:46.335417 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:46.335375    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"]
Sep 16 08:06:46.335868 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:46.335718    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9" podUID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerName="istio-proxy" containerID="cri-o://c78fbc68cd108c65f87ef7def89140b728e66e4fa7489a119cb6b7f2d6f2da70" gracePeriod=30
Sep 16 08:06:50.101422 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:50.101379    2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 08:06:50.101815 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:50.101448    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9" podUID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 08:06:50.205105 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:50.205068    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6"]
Sep 16 08:06:50.205549 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:50.205524    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e464e515-e843-43b6-a086-10e90ceb6c6d" containerName="payload-processing"
Sep 16 08:06:50.205549 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:50.205542    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e464e515-e843-43b6-a086-10e90ceb6c6d" containerName="payload-processing"
Sep 16 08:06:50.205757 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:50.205622    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e464e515-e843-43b6-a086-10e90ceb6c6d" containerName="payload-processing"
Sep 16 08:06:50.208667 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:50.208648    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6"
Sep 16 08:06:50.210890 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:50.210868    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\""
Sep 16 08:06:50.211010 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:50.210892    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 08:06:50.218481 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:50.218121    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6"]
Sep 16 08:06:50.336750 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:50.336704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6\" (UID: \"b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6"
Sep 16 08:06:50.442060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:50.440977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6\" (UID: \"b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6"
Sep 16 08:06:50.442060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:50.442009    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6\" (UID: \"b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6"
Sep 16 08:06:50.522718 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:50.522678    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6"
Sep 16 08:06:50.793335 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:50.793303    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6"]
Sep 16 08:06:50.794713 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:06:50.794681    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b8b4ec_49e0_4052_b56a_e90d1c3e4c99.slice/crio-6ab813034d04e71a2eae3a57a39741615ce429c193eaec43aabd98e6b842e220 WatchSource:0}: Error finding container 6ab813034d04e71a2eae3a57a39741615ce429c193eaec43aabd98e6b842e220: Status 404 returned error can't find the container with id 6ab813034d04e71a2eae3a57a39741615ce429c193eaec43aabd98e6b842e220
Sep 16 08:06:51.225683 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.225645    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6" event={"ID":"b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99","Type":"ContainerStarted","Data":"3eae87f0750400df4d4b13343ebaac63dc08e5899d8f53f98a7ea4c1d6a598cc"}
Sep 16 08:06:51.226095 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.225700    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6" event={"ID":"b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99","Type":"ContainerStarted","Data":"6ab813034d04e71a2eae3a57a39741615ce429c193eaec43aabd98e6b842e220"}
Sep 16 08:06:51.241830 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.241773    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6" podStartSLOduration=1.241756052 podStartE2EDuration="1.241756052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:06:51.239744327 +0000 UTC m=+1972.166356480" watchObservedRunningTime="2026-09-16 08:06:51.241756052 +0000 UTC m=+1972.168368206"
Sep 16 08:06:51.582211 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.582187    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:51.655077 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655040    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204440a6-7b16-4780-8c29-37d3bd4bdd29-istiod-ca-cert\") pod \"204440a6-7b16-4780-8c29-37d3bd4bdd29\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") "
Sep 16 08:06:51.655271 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655150    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-workload-socket\") pod \"204440a6-7b16-4780-8c29-37d3bd4bdd29\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") "
Sep 16 08:06:51.655271 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655190    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-envoy\") pod \"204440a6-7b16-4780-8c29-37d3bd4bdd29\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") "
Sep 16 08:06:51.655271 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655251    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d7z5\" (UniqueName: \"kubernetes.io/projected/204440a6-7b16-4780-8c29-37d3bd4bdd29-kube-api-access-6d7z5\") pod \"204440a6-7b16-4780-8c29-37d3bd4bdd29\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") "
Sep 16 08:06:51.655463 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655288    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-data\") pod \"204440a6-7b16-4780-8c29-37d3bd4bdd29\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") "
Sep 16 08:06:51.655463 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655314    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-podinfo\") pod \"204440a6-7b16-4780-8c29-37d3bd4bdd29\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") "
Sep 16 08:06:51.655463 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655341    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-token\") pod \"204440a6-7b16-4780-8c29-37d3bd4bdd29\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") "
Sep 16 08:06:51.655463 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655412    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-credential-socket\") pod \"204440a6-7b16-4780-8c29-37d3bd4bdd29\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") "
Sep 16 08:06:51.655659 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655471    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-workload-certs\") pod \"204440a6-7b16-4780-8c29-37d3bd4bdd29\" (UID: \"204440a6-7b16-4780-8c29-37d3bd4bdd29\") "
Sep 16 08:06:51.655659 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655473    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "204440a6-7b16-4780-8c29-37d3bd4bdd29" (UID: "204440a6-7b16-4780-8c29-37d3bd4bdd29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:06:51.655659 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655566    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204440a6-7b16-4780-8c29-37d3bd4bdd29-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "204440a6-7b16-4780-8c29-37d3bd4bdd29" (UID: "204440a6-7b16-4780-8c29-37d3bd4bdd29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:06:51.655659 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655569    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-data" (OuterVolumeSpecName: "istio-data") pod "204440a6-7b16-4780-8c29-37d3bd4bdd29" (UID: "204440a6-7b16-4780-8c29-37d3bd4bdd29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:06:51.655906 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655715    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "204440a6-7b16-4780-8c29-37d3bd4bdd29" (UID: "204440a6-7b16-4780-8c29-37d3bd4bdd29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:06:51.655906 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655783    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "204440a6-7b16-4780-8c29-37d3bd4bdd29" (UID: "204440a6-7b16-4780-8c29-37d3bd4bdd29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:06:51.655906 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655861    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204440a6-7b16-4780-8c29-37d3bd4bdd29-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:06:51.655906 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655880    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:06:51.655906 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655894    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:06:51.656083 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.655907    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:06:51.657524 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.657495    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "204440a6-7b16-4780-8c29-37d3bd4bdd29" (UID: "204440a6-7b16-4780-8c29-37d3bd4bdd29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:06:51.657636 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.657617    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204440a6-7b16-4780-8c29-37d3bd4bdd29-kube-api-access-6d7z5" (OuterVolumeSpecName: "kube-api-access-6d7z5") pod "204440a6-7b16-4780-8c29-37d3bd4bdd29" (UID: "204440a6-7b16-4780-8c29-37d3bd4bdd29"). InnerVolumeSpecName "kube-api-access-6d7z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:06:51.657993 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.657970    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-token" (OuterVolumeSpecName: "istio-token") pod "204440a6-7b16-4780-8c29-37d3bd4bdd29" (UID: "204440a6-7b16-4780-8c29-37d3bd4bdd29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:06:51.657993 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.657978    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "204440a6-7b16-4780-8c29-37d3bd4bdd29" (UID: "204440a6-7b16-4780-8c29-37d3bd4bdd29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 08:06:51.757450 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.757347    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:06:51.757450 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.757380    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d7z5\" (UniqueName: \"kubernetes.io/projected/204440a6-7b16-4780-8c29-37d3bd4bdd29-kube-api-access-6d7z5\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:06:51.757450 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.757389    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:06:51.757450 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.757397    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204440a6-7b16-4780-8c29-37d3bd4bdd29-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:06:51.757450 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:51.757410    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204440a6-7b16-4780-8c29-37d3bd4bdd29-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:06:52.232126 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:52.232085    2581 generic.go:356] "Generic (PLEG): container finished" podID="b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99" containerID="3eae87f0750400df4d4b13343ebaac63dc08e5899d8f53f98a7ea4c1d6a598cc" exitCode=0
Sep 16 08:06:52.232554 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:52.232177    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6" event={"ID":"b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99","Type":"ContainerDied","Data":"3eae87f0750400df4d4b13343ebaac63dc08e5899d8f53f98a7ea4c1d6a598cc"}
Sep 16 08:06:52.233877 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:52.233845    2581 generic.go:356] "Generic (PLEG): container finished" podID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerID="c78fbc68cd108c65f87ef7def89140b728e66e4fa7489a119cb6b7f2d6f2da70" exitCode=0
Sep 16 08:06:52.233996 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:52.233874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9" event={"ID":"204440a6-7b16-4780-8c29-37d3bd4bdd29","Type":"ContainerDied","Data":"c78fbc68cd108c65f87ef7def89140b728e66e4fa7489a119cb6b7f2d6f2da70"}
Sep 16 08:06:52.233996 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:52.233900    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"
Sep 16 08:06:52.233996 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:52.233924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9" event={"ID":"204440a6-7b16-4780-8c29-37d3bd4bdd29","Type":"ContainerDied","Data":"7ded03bc567f000f7f888c9c0aed0f233d21133b2fbe368a936b0e16a5e65a14"}
Sep 16 08:06:52.233996 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:52.233950    2581 scope.go:117] "RemoveContainer" containerID="c78fbc68cd108c65f87ef7def89140b728e66e4fa7489a119cb6b7f2d6f2da70"
Sep 16 08:06:52.242845 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:52.242814    2581 scope.go:117] "RemoveContainer" containerID="313b2944d904ce3e3c224c0473095022645780436b30920dbd79138bdad4724d"
Sep 16 08:06:52.251858 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:52.251836    2581 scope.go:117] "RemoveContainer" containerID="c78fbc68cd108c65f87ef7def89140b728e66e4fa7489a119cb6b7f2d6f2da70"
Sep 16 08:06:52.252135 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:06:52.252116    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c78fbc68cd108c65f87ef7def89140b728e66e4fa7489a119cb6b7f2d6f2da70\": container with ID starting with c78fbc68cd108c65f87ef7def89140b728e66e4fa7489a119cb6b7f2d6f2da70 not found: ID does not exist" containerID="c78fbc68cd108c65f87ef7def89140b728e66e4fa7489a119cb6b7f2d6f2da70"
Sep 16 08:06:52.252216 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:52.252143    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78fbc68cd108c65f87ef7def89140b728e66e4fa7489a119cb6b7f2d6f2da70"} err="failed to get container status \"c78fbc68cd108c65f87ef7def89140b728e66e4fa7489a119cb6b7f2d6f2da70\": rpc error: code = NotFound desc = could not find container \"c78fbc68cd108c65f87ef7def89140b728e66e4fa7489a119cb6b7f2d6f2da70\": container with ID starting with c78fbc68cd108c65f87ef7def89140b728e66e4fa7489a119cb6b7f2d6f2da70 not found: ID does not exist"
Sep 16 08:06:52.252216 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:52.252159    2581 scope.go:117] "RemoveContainer" containerID="313b2944d904ce3e3c224c0473095022645780436b30920dbd79138bdad4724d"
Sep 16 08:06:52.252544 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:06:52.252516    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313b2944d904ce3e3c224c0473095022645780436b30920dbd79138bdad4724d\": container with ID starting with 313b2944d904ce3e3c224c0473095022645780436b30920dbd79138bdad4724d not found: ID does not exist" containerID="313b2944d904ce3e3c224c0473095022645780436b30920dbd79138bdad4724d"
Sep 16 08:06:52.252638 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:52.252550    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313b2944d904ce3e3c224c0473095022645780436b30920dbd79138bdad4724d"} err="failed to get container status \"313b2944d904ce3e3c224c0473095022645780436b30920dbd79138bdad4724d\": rpc error: code = NotFound desc = could not find container \"313b2944d904ce3e3c224c0473095022645780436b30920dbd79138bdad4724d\": container with ID starting with 313b2944d904ce3e3c224c0473095022645780436b30920dbd79138bdad4724d not found: ID does not exist"
Sep 16 08:06:52.263365 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:52.263340    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"]
Sep 16 08:06:52.267007 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:52.266985    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d26a85-openshift-default-c76fd8c88-4vzb9"]
Sep 16 08:06:53.381521 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:53.381497    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6"
Sep 16 08:06:53.475895 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:53.475854    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99-maas-api-service-ca\") pod \"b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99\" (UID: \"b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99\") "
Sep 16 08:06:53.476247 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:53.476224    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99" (UID: "b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:06:53.576991 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:53.576901    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:06:53.684283 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:53.684241    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="204440a6-7b16-4780-8c29-37d3bd4bdd29" path="/var/lib/kubelet/pods/204440a6-7b16-4780-8c29-37d3bd4bdd29/volumes"
Sep 16 08:06:54.247351 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:54.247321    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6"
Sep 16 08:06:54.247547 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:54.247315    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6" event={"ID":"b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99","Type":"ContainerDied","Data":"6ab813034d04e71a2eae3a57a39741615ce429c193eaec43aabd98e6b842e220"}
Sep 16 08:06:54.247547 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:54.247463    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ab813034d04e71a2eae3a57a39741615ce429c193eaec43aabd98e6b842e220"
Sep 16 08:06:59.986934 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:59.986902    2581 scope.go:117] "RemoveContainer" containerID="24e15325d5ae77cc62478889348c5412b8a9219344062aa2cf3cf3052569bc55"
Sep 16 08:06:59.995378 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:06:59.995356    2581 scope.go:117] "RemoveContainer" containerID="db2357103f5464668bd7f4160d12b60eab4cfa328d7704843210d95217913514"
Sep 16 08:07:05.725367 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:05.725323    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"]
Sep 16 08:07:05.726067 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:05.725613    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p" podUID="1f16370a-811e-4d4c-97d8-2be259b73b5a" containerName="payload-processing" containerID="cri-o://85d066aa1d3f0d3e5cfae4b8183daade58a54601a3df5ff63c968adef13b5ab3" gracePeriod=30
Sep 16 08:07:05.730712 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:05.730672    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"]
Sep 16 08:07:05.731192 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:05.731144    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn" podUID="fc41361f-cd2b-45b1-a90f-8e3858676592" containerName="payload-pre-processing" containerID="cri-o://442be6aa205886a8ac253fda3f3ac4adebb310af6c26600b9c3449a839bbbbbe" gracePeriod=30
Sep 16 08:07:05.883513 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:05.883442    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6"]
Sep 16 08:07:05.885525 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:05.885497    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-796e11-z2qv6"]
Sep 16 08:07:06.032974 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.032941    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"
Sep 16 08:07:06.036107 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.036084    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"
Sep 16 08:07:06.102537 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.102501    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mmlv\" (UniqueName: \"kubernetes.io/projected/1f16370a-811e-4d4c-97d8-2be259b73b5a-kube-api-access-2mmlv\") pod \"1f16370a-811e-4d4c-97d8-2be259b73b5a\" (UID: \"1f16370a-811e-4d4c-97d8-2be259b73b5a\") "
Sep 16 08:07:06.102537 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.102541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc41361f-cd2b-45b1-a90f-8e3858676592-plugins-config-volume\") pod \"fc41361f-cd2b-45b1-a90f-8e3858676592\" (UID: \"fc41361f-cd2b-45b1-a90f-8e3858676592\") "
Sep 16 08:07:06.102803 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.102602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f16370a-811e-4d4c-97d8-2be259b73b5a-plugins-config-volume\") pod \"1f16370a-811e-4d4c-97d8-2be259b73b5a\" (UID: \"1f16370a-811e-4d4c-97d8-2be259b73b5a\") "
Sep 16 08:07:06.102803 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.102694    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj5fx\" (UniqueName: \"kubernetes.io/projected/fc41361f-cd2b-45b1-a90f-8e3858676592-kube-api-access-wj5fx\") pod \"fc41361f-cd2b-45b1-a90f-8e3858676592\" (UID: \"fc41361f-cd2b-45b1-a90f-8e3858676592\") "
Sep 16 08:07:06.102963 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.102941    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f16370a-811e-4d4c-97d8-2be259b73b5a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1f16370a-811e-4d4c-97d8-2be259b73b5a" (UID: "1f16370a-811e-4d4c-97d8-2be259b73b5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:07:06.103015 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.102966    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc41361f-cd2b-45b1-a90f-8e3858676592-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc41361f-cd2b-45b1-a90f-8e3858676592" (UID: "fc41361f-cd2b-45b1-a90f-8e3858676592"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:07:06.103128 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.103114    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f16370a-811e-4d4c-97d8-2be259b73b5a-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:07:06.103173 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.103134    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc41361f-cd2b-45b1-a90f-8e3858676592-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:07:06.104723 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.104690    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc41361f-cd2b-45b1-a90f-8e3858676592-kube-api-access-wj5fx" (OuterVolumeSpecName: "kube-api-access-wj5fx") pod "fc41361f-cd2b-45b1-a90f-8e3858676592" (UID: "fc41361f-cd2b-45b1-a90f-8e3858676592"). InnerVolumeSpecName "kube-api-access-wj5fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:07:06.104853 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.104783    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f16370a-811e-4d4c-97d8-2be259b73b5a-kube-api-access-2mmlv" (OuterVolumeSpecName: "kube-api-access-2mmlv") pod "1f16370a-811e-4d4c-97d8-2be259b73b5a" (UID: "1f16370a-811e-4d4c-97d8-2be259b73b5a"). InnerVolumeSpecName "kube-api-access-2mmlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:07:06.204107 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.204068    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mmlv\" (UniqueName: \"kubernetes.io/projected/1f16370a-811e-4d4c-97d8-2be259b73b5a-kube-api-access-2mmlv\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:07:06.204107 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.204098    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj5fx\" (UniqueName: \"kubernetes.io/projected/fc41361f-cd2b-45b1-a90f-8e3858676592-kube-api-access-wj5fx\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:07:06.311075 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.310984    2581 generic.go:356] "Generic (PLEG): container finished" podID="1f16370a-811e-4d4c-97d8-2be259b73b5a" containerID="85d066aa1d3f0d3e5cfae4b8183daade58a54601a3df5ff63c968adef13b5ab3" exitCode=0
Sep 16 08:07:06.311075 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.311029    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"
Sep 16 08:07:06.311309 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.311068    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p" event={"ID":"1f16370a-811e-4d4c-97d8-2be259b73b5a","Type":"ContainerDied","Data":"85d066aa1d3f0d3e5cfae4b8183daade58a54601a3df5ff63c968adef13b5ab3"}
Sep 16 08:07:06.311309 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.311129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p" event={"ID":"1f16370a-811e-4d4c-97d8-2be259b73b5a","Type":"ContainerDied","Data":"60d6c724580897bc306f45ec24239fe560f3df9ef8699088484691f30dfb5f95"}
Sep 16 08:07:06.311309 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.311157    2581 scope.go:117] "RemoveContainer" containerID="85d066aa1d3f0d3e5cfae4b8183daade58a54601a3df5ff63c968adef13b5ab3"
Sep 16 08:07:06.312933 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.312903    2581 generic.go:356] "Generic (PLEG): container finished" podID="fc41361f-cd2b-45b1-a90f-8e3858676592" containerID="442be6aa205886a8ac253fda3f3ac4adebb310af6c26600b9c3449a839bbbbbe" exitCode=0
Sep 16 08:07:06.313072 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.312947    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"
Sep 16 08:07:06.313072 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.313049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn" event={"ID":"fc41361f-cd2b-45b1-a90f-8e3858676592","Type":"ContainerDied","Data":"442be6aa205886a8ac253fda3f3ac4adebb310af6c26600b9c3449a839bbbbbe"}
Sep 16 08:07:06.313232 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.313095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn" event={"ID":"fc41361f-cd2b-45b1-a90f-8e3858676592","Type":"ContainerDied","Data":"26a0fd6ccceaf49accab7d659e43bbc97d251e37ac0739088df94148f7c64787"}
Sep 16 08:07:06.322150 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.322120    2581 scope.go:117] "RemoveContainer" containerID="85d066aa1d3f0d3e5cfae4b8183daade58a54601a3df5ff63c968adef13b5ab3"
Sep 16 08:07:06.322518 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:07:06.322492    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d066aa1d3f0d3e5cfae4b8183daade58a54601a3df5ff63c968adef13b5ab3\": container with ID starting with 85d066aa1d3f0d3e5cfae4b8183daade58a54601a3df5ff63c968adef13b5ab3 not found: ID does not exist" containerID="85d066aa1d3f0d3e5cfae4b8183daade58a54601a3df5ff63c968adef13b5ab3"
Sep 16 08:07:06.322635 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.322531    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d066aa1d3f0d3e5cfae4b8183daade58a54601a3df5ff63c968adef13b5ab3"} err="failed to get container status \"85d066aa1d3f0d3e5cfae4b8183daade58a54601a3df5ff63c968adef13b5ab3\": rpc error: code = NotFound desc = could not find container \"85d066aa1d3f0d3e5cfae4b8183daade58a54601a3df5ff63c968adef13b5ab3\": container with ID starting with 85d066aa1d3f0d3e5cfae4b8183daade58a54601a3df5ff63c968adef13b5ab3 not found: ID does not exist"
Sep 16 08:07:06.322635 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.322558    2581 scope.go:117] "RemoveContainer" containerID="442be6aa205886a8ac253fda3f3ac4adebb310af6c26600b9c3449a839bbbbbe"
Sep 16 08:07:06.331322 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.331293    2581 scope.go:117] "RemoveContainer" containerID="442be6aa205886a8ac253fda3f3ac4adebb310af6c26600b9c3449a839bbbbbe"
Sep 16 08:07:06.331631 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:07:06.331611    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442be6aa205886a8ac253fda3f3ac4adebb310af6c26600b9c3449a839bbbbbe\": container with ID starting with 442be6aa205886a8ac253fda3f3ac4adebb310af6c26600b9c3449a839bbbbbe not found: ID does not exist" containerID="442be6aa205886a8ac253fda3f3ac4adebb310af6c26600b9c3449a839bbbbbe"
Sep 16 08:07:06.331682 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.331656    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442be6aa205886a8ac253fda3f3ac4adebb310af6c26600b9c3449a839bbbbbe"} err="failed to get container status \"442be6aa205886a8ac253fda3f3ac4adebb310af6c26600b9c3449a839bbbbbe\": rpc error: code = NotFound desc = could not find container \"442be6aa205886a8ac253fda3f3ac4adebb310af6c26600b9c3449a839bbbbbe\": container with ID starting with 442be6aa205886a8ac253fda3f3ac4adebb310af6c26600b9c3449a839bbbbbe not found: ID does not exist"
Sep 16 08:07:06.341451 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.341405    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"]
Sep 16 08:07:06.349502 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.349469    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-796e11-57b495bf95-mrb4p"]
Sep 16 08:07:06.359060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.359013    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"]
Sep 16 08:07:06.365672 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:06.365638    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-796e11-5cf89b8498-jxgnn"]
Sep 16 08:07:07.684459 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:07.684417    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f16370a-811e-4d4c-97d8-2be259b73b5a" path="/var/lib/kubelet/pods/1f16370a-811e-4d4c-97d8-2be259b73b5a/volumes"
Sep 16 08:07:07.684978 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:07.684923    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99" path="/var/lib/kubelet/pods/b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99/volumes"
Sep 16 08:07:07.685324 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:07.685304    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc41361f-cd2b-45b1-a90f-8e3858676592" path="/var/lib/kubelet/pods/fc41361f-cd2b-45b1-a90f-8e3858676592/volumes"
Sep 16 08:07:54.123952 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.123915    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"]
Sep 16 08:07:54.124400 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.124139    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx" podUID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" containerName="istio-proxy" containerID="cri-o://679ac30484983e0e21fabfe37f25836f2db7f6d3a507d3b827299cf5e4d0fcc0" gracePeriod=30
Sep 16 08:07:54.481123 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.481087    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct"]
Sep 16 08:07:54.481498 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.481486    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc41361f-cd2b-45b1-a90f-8e3858676592" containerName="payload-pre-processing"
Sep 16 08:07:54.481545 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.481500    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc41361f-cd2b-45b1-a90f-8e3858676592" containerName="payload-pre-processing"
Sep 16 08:07:54.481545 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.481509    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerName="istio-proxy"
Sep 16 08:07:54.481545 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.481516    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerName="istio-proxy"
Sep 16 08:07:54.481545 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.481532    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f16370a-811e-4d4c-97d8-2be259b73b5a" containerName="payload-processing"
Sep 16 08:07:54.481545 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.481539    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f16370a-811e-4d4c-97d8-2be259b73b5a" containerName="payload-processing"
Sep 16 08:07:54.481707 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.481550    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99" containerName="revoke-keys"
Sep 16 08:07:54.481707 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.481556    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99" containerName="revoke-keys"
Sep 16 08:07:54.481707 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.481608    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc41361f-cd2b-45b1-a90f-8e3858676592" containerName="payload-pre-processing"
Sep 16 08:07:54.481707 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.481615    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f16370a-811e-4d4c-97d8-2be259b73b5a" containerName="payload-processing"
Sep 16 08:07:54.481707 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.481622    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b8b4ec-49e0-4052-b56a-e90d1c3e4c99" containerName="revoke-keys"
Sep 16 08:07:54.481707 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.481627    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerName="istio-proxy"
Sep 16 08:07:54.481707 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.481634    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerName="istio-proxy"
Sep 16 08:07:54.484665 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.484647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct"
Sep 16 08:07:54.487184 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.487154    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\""
Sep 16 08:07:54.487184 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.487176    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 08:07:54.493366 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.493339    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct"]
Sep 16 08:07:54.559948 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.559915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcd7a2d-625a-4c2e-9722-8f02d98188a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct\" (UID: \"8bcd7a2d-625a-4c2e-9722-8f02d98188a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct"
Sep 16 08:07:54.660660 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.660620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcd7a2d-625a-4c2e-9722-8f02d98188a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct\" (UID: \"8bcd7a2d-625a-4c2e-9722-8f02d98188a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct"
Sep 16 08:07:54.661296 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.661277    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcd7a2d-625a-4c2e-9722-8f02d98188a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct\" (UID: \"8bcd7a2d-625a-4c2e-9722-8f02d98188a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct"
Sep 16 08:07:54.796929 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:54.796807    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct"
Sep 16 08:07:55.068452 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:55.068427    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct"]
Sep 16 08:07:55.070196 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:07:55.070170    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcd7a2d_625a_4c2e_9722_8f02d98188a6.slice/crio-3cc5f42310522405ccdd9eb282ab6c18442e0047fd3765d17f7d41d7f9f76149 WatchSource:0}: Error finding container 3cc5f42310522405ccdd9eb282ab6c18442e0047fd3765d17f7d41d7f9f76149: Status 404 returned error can't find the container with id 3cc5f42310522405ccdd9eb282ab6c18442e0047fd3765d17f7d41d7f9f76149
Sep 16 08:07:55.570426 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:55.570392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct" event={"ID":"8bcd7a2d-625a-4c2e-9722-8f02d98188a6","Type":"ContainerStarted","Data":"95c958f9e7ecb3f98a860e07bdb176a468b358c7a0aa352f0f6737603ae39f50"}
Sep 16 08:07:55.570875 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:55.570441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct" event={"ID":"8bcd7a2d-625a-4c2e-9722-8f02d98188a6","Type":"ContainerStarted","Data":"3cc5f42310522405ccdd9eb282ab6c18442e0047fd3765d17f7d41d7f9f76149"}
Sep 16 08:07:55.588171 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:55.588071    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct" podStartSLOduration=1.5880540600000002 podStartE2EDuration="1.58805406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:07:55.587068775 +0000 UTC m=+2036.513680930" watchObservedRunningTime="2026-09-16 08:07:55.58805406 +0000 UTC m=+2036.514666212"
Sep 16 08:07:56.576895 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:56.576863    2581 generic.go:356] "Generic (PLEG): container finished" podID="8bcd7a2d-625a-4c2e-9722-8f02d98188a6" containerID="95c958f9e7ecb3f98a860e07bdb176a468b358c7a0aa352f0f6737603ae39f50" exitCode=0
Sep 16 08:07:56.577290 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:56.576905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct" event={"ID":"8bcd7a2d-625a-4c2e-9722-8f02d98188a6","Type":"ContainerDied","Data":"95c958f9e7ecb3f98a860e07bdb176a468b358c7a0aa352f0f6737603ae39f50"}
Sep 16 08:07:57.717131 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:57.717105    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct"
Sep 16 08:07:57.793529 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:57.793494    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcd7a2d-625a-4c2e-9722-8f02d98188a6-maas-api-service-ca\") pod \"8bcd7a2d-625a-4c2e-9722-8f02d98188a6\" (UID: \"8bcd7a2d-625a-4c2e-9722-8f02d98188a6\") "
Sep 16 08:07:57.793873 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:57.793813    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcd7a2d-625a-4c2e-9722-8f02d98188a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8bcd7a2d-625a-4c2e-9722-8f02d98188a6" (UID: "8bcd7a2d-625a-4c2e-9722-8f02d98188a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:07:57.895288 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:57.895197    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcd7a2d-625a-4c2e-9722-8f02d98188a6-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:07:58.587551 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:58.587524    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct"
Sep 16 08:07:58.587741 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:58.587528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct" event={"ID":"8bcd7a2d-625a-4c2e-9722-8f02d98188a6","Type":"ContainerDied","Data":"3cc5f42310522405ccdd9eb282ab6c18442e0047fd3765d17f7d41d7f9f76149"}
Sep 16 08:07:58.587741 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:58.587647    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cc5f42310522405ccdd9eb282ab6c18442e0047fd3765d17f7d41d7f9f76149"
Sep 16 08:07:59.392460 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.392389    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:07:59.513638 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.513606    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-workload-certs\") pod \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") "
Sep 16 08:07:59.513851 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.513644    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-podinfo\") pod \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") "
Sep 16 08:07:59.513851 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.513674    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-credential-socket\") pod \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") "
Sep 16 08:07:59.513851 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.513698    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-envoy\") pod \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") "
Sep 16 08:07:59.513851 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.513721    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istiod-ca-cert\") pod \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") "
Sep 16 08:07:59.513851 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.513747    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-data\") pod \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") "
Sep 16 08:07:59.513851 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.513779    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-workload-socket\") pod \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") "
Sep 16 08:07:59.514168 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.513894    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxfzn\" (UniqueName: \"kubernetes.io/projected/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-kube-api-access-lxfzn\") pod \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") "
Sep 16 08:07:59.514168 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.513947    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-token\") pod \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\" (UID: \"f3fd96c9-b483-44a9-9530-65f20cc8bfe3\") "
Sep 16 08:07:59.514168 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.513941    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f3fd96c9-b483-44a9-9530-65f20cc8bfe3" (UID: "f3fd96c9-b483-44a9-9530-65f20cc8bfe3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:07:59.514168 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.514002    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f3fd96c9-b483-44a9-9530-65f20cc8bfe3" (UID: "f3fd96c9-b483-44a9-9530-65f20cc8bfe3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:07:59.514168 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.514130    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f3fd96c9-b483-44a9-9530-65f20cc8bfe3" (UID: "f3fd96c9-b483-44a9-9530-65f20cc8bfe3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:07:59.514437 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.514216    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f3fd96c9-b483-44a9-9530-65f20cc8bfe3" (UID: "f3fd96c9-b483-44a9-9530-65f20cc8bfe3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:07:59.514437 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.514302    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:07:59.514437 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.514320    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:07:59.514437 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.514334    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:07:59.514437 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.514347    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:07:59.514437 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.514426    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-data" (OuterVolumeSpecName: "istio-data") pod "f3fd96c9-b483-44a9-9530-65f20cc8bfe3" (UID: "f3fd96c9-b483-44a9-9530-65f20cc8bfe3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:07:59.516081 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.516060    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f3fd96c9-b483-44a9-9530-65f20cc8bfe3" (UID: "f3fd96c9-b483-44a9-9530-65f20cc8bfe3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 08:07:59.516154 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.516121    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f3fd96c9-b483-44a9-9530-65f20cc8bfe3" (UID: "f3fd96c9-b483-44a9-9530-65f20cc8bfe3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:07:59.516443 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.516423    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-token" (OuterVolumeSpecName: "istio-token") pod "f3fd96c9-b483-44a9-9530-65f20cc8bfe3" (UID: "f3fd96c9-b483-44a9-9530-65f20cc8bfe3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:07:59.516443 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.516436    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-kube-api-access-lxfzn" (OuterVolumeSpecName: "kube-api-access-lxfzn") pod "f3fd96c9-b483-44a9-9530-65f20cc8bfe3" (UID: "f3fd96c9-b483-44a9-9530-65f20cc8bfe3"). InnerVolumeSpecName "kube-api-access-lxfzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:07:59.601231 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.601197    2581 generic.go:356] "Generic (PLEG): container finished" podID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" containerID="679ac30484983e0e21fabfe37f25836f2db7f6d3a507d3b827299cf5e4d0fcc0" exitCode=0
Sep 16 08:07:59.601418 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.601257    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"
Sep 16 08:07:59.601418 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.601278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx" event={"ID":"f3fd96c9-b483-44a9-9530-65f20cc8bfe3","Type":"ContainerDied","Data":"679ac30484983e0e21fabfe37f25836f2db7f6d3a507d3b827299cf5e4d0fcc0"}
Sep 16 08:07:59.601418 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.601353    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx" event={"ID":"f3fd96c9-b483-44a9-9530-65f20cc8bfe3","Type":"ContainerDied","Data":"7178ab65d53ec2d3a8a14ea50ddccc9458aa4e02cfbb808f8805c39e2c8e8951"}
Sep 16 08:07:59.601418 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.601376    2581 scope.go:117] "RemoveContainer" containerID="679ac30484983e0e21fabfe37f25836f2db7f6d3a507d3b827299cf5e4d0fcc0"
Sep 16 08:07:59.611529 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.611509    2581 scope.go:117] "RemoveContainer" containerID="24c1ca9d0679669bc7521e7aacb969efddf83c6686f1bd371cb1469476a429d6"
Sep 16 08:07:59.615753 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.615719    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxfzn\" (UniqueName: \"kubernetes.io/projected/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-kube-api-access-lxfzn\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:07:59.615753 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.615750    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:07:59.615993 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.615763    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:07:59.615993 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.615772    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:07:59.615993 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.615781    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3fd96c9-b483-44a9-9530-65f20cc8bfe3-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:07:59.621192 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.621162    2581 scope.go:117] "RemoveContainer" containerID="679ac30484983e0e21fabfe37f25836f2db7f6d3a507d3b827299cf5e4d0fcc0"
Sep 16 08:07:59.621778 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:07:59.621741    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"679ac30484983e0e21fabfe37f25836f2db7f6d3a507d3b827299cf5e4d0fcc0\": container with ID starting with 679ac30484983e0e21fabfe37f25836f2db7f6d3a507d3b827299cf5e4d0fcc0 not found: ID does not exist" containerID="679ac30484983e0e21fabfe37f25836f2db7f6d3a507d3b827299cf5e4d0fcc0"
Sep 16 08:07:59.621977 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.621792    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"679ac30484983e0e21fabfe37f25836f2db7f6d3a507d3b827299cf5e4d0fcc0"} err="failed to get container status \"679ac30484983e0e21fabfe37f25836f2db7f6d3a507d3b827299cf5e4d0fcc0\": rpc error: code = NotFound desc = could not find container \"679ac30484983e0e21fabfe37f25836f2db7f6d3a507d3b827299cf5e4d0fcc0\": container with ID starting with 679ac30484983e0e21fabfe37f25836f2db7f6d3a507d3b827299cf5e4d0fcc0 not found: ID does not exist"
Sep 16 08:07:59.621977 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.621834    2581 scope.go:117] "RemoveContainer" containerID="24c1ca9d0679669bc7521e7aacb969efddf83c6686f1bd371cb1469476a429d6"
Sep 16 08:07:59.622232 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:07:59.622209    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c1ca9d0679669bc7521e7aacb969efddf83c6686f1bd371cb1469476a429d6\": container with ID starting with 24c1ca9d0679669bc7521e7aacb969efddf83c6686f1bd371cb1469476a429d6 not found: ID does not exist" containerID="24c1ca9d0679669bc7521e7aacb969efddf83c6686f1bd371cb1469476a429d6"
Sep 16 08:07:59.622306 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.622243    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c1ca9d0679669bc7521e7aacb969efddf83c6686f1bd371cb1469476a429d6"} err="failed to get container status \"24c1ca9d0679669bc7521e7aacb969efddf83c6686f1bd371cb1469476a429d6\": rpc error: code = NotFound desc = could not find container \"24c1ca9d0679669bc7521e7aacb969efddf83c6686f1bd371cb1469476a429d6\": container with ID starting with 24c1ca9d0679669bc7521e7aacb969efddf83c6686f1bd371cb1469476a429d6 not found: ID does not exist"
Sep 16 08:07:59.624114 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.624093    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"]
Sep 16 08:07:59.627556 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.627531    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-796e11-openshift-default-659fd45f5b-xm5kx"]
Sep 16 08:07:59.684418 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:07:59.684391    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" path="/var/lib/kubelet/pods/f3fd96c9-b483-44a9-9530-65f20cc8bfe3/volumes"
Sep 16 08:08:04.942990 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:04.942954    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"]
Sep 16 08:08:04.943442 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:04.943216    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5" podUID="f47dd702-1136-4b91-b42c-b9153b93fc49" containerName="payload-processing" containerID="cri-o://462ac42ee413df3a7a999bfd805a28348f76289da4b8530d0e19f1efd7260aeb" gracePeriod=30
Sep 16 08:08:04.948807 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:04.948778    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"]
Sep 16 08:08:04.949286 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:04.949248    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp" podUID="460d7a26-c5fe-47b0-8a5e-bcc6e318ac44" containerName="payload-pre-processing" containerID="cri-o://378e57ab8e0e1c8cc5f2be3ad1ab2ea0e0bcf35589325ce0c5a7d56d2b3eaa42" gracePeriod=30
Sep 16 08:08:05.095936 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.095861    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct"]
Sep 16 08:08:05.097505 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.097479    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5493e8-tzwct"]
Sep 16 08:08:05.234100 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.234069    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"
Sep 16 08:08:05.237236 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.237213    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"
Sep 16 08:08:05.368564 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.368525    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460d7a26-c5fe-47b0-8a5e-bcc6e318ac44-plugins-config-volume\") pod \"460d7a26-c5fe-47b0-8a5e-bcc6e318ac44\" (UID: \"460d7a26-c5fe-47b0-8a5e-bcc6e318ac44\") "
Sep 16 08:08:05.368748 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.368580    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz7hf\" (UniqueName: \"kubernetes.io/projected/f47dd702-1136-4b91-b42c-b9153b93fc49-kube-api-access-xz7hf\") pod \"f47dd702-1136-4b91-b42c-b9153b93fc49\" (UID: \"f47dd702-1136-4b91-b42c-b9153b93fc49\") "
Sep 16 08:08:05.368748 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.368612    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsxnh\" (UniqueName: \"kubernetes.io/projected/460d7a26-c5fe-47b0-8a5e-bcc6e318ac44-kube-api-access-qsxnh\") pod \"460d7a26-c5fe-47b0-8a5e-bcc6e318ac44\" (UID: \"460d7a26-c5fe-47b0-8a5e-bcc6e318ac44\") "
Sep 16 08:08:05.368748 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.368697    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47dd702-1136-4b91-b42c-b9153b93fc49-plugins-config-volume\") pod \"f47dd702-1136-4b91-b42c-b9153b93fc49\" (UID: \"f47dd702-1136-4b91-b42c-b9153b93fc49\") "
Sep 16 08:08:05.369019 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.368991    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460d7a26-c5fe-47b0-8a5e-bcc6e318ac44-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "460d7a26-c5fe-47b0-8a5e-bcc6e318ac44" (UID: "460d7a26-c5fe-47b0-8a5e-bcc6e318ac44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:08:05.369165 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.369139    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47dd702-1136-4b91-b42c-b9153b93fc49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f47dd702-1136-4b91-b42c-b9153b93fc49" (UID: "f47dd702-1136-4b91-b42c-b9153b93fc49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:08:05.370850 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.370793    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47dd702-1136-4b91-b42c-b9153b93fc49-kube-api-access-xz7hf" (OuterVolumeSpecName: "kube-api-access-xz7hf") pod "f47dd702-1136-4b91-b42c-b9153b93fc49" (UID: "f47dd702-1136-4b91-b42c-b9153b93fc49"). InnerVolumeSpecName "kube-api-access-xz7hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:08:05.370850 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.370798    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460d7a26-c5fe-47b0-8a5e-bcc6e318ac44-kube-api-access-qsxnh" (OuterVolumeSpecName: "kube-api-access-qsxnh") pod "460d7a26-c5fe-47b0-8a5e-bcc6e318ac44" (UID: "460d7a26-c5fe-47b0-8a5e-bcc6e318ac44"). InnerVolumeSpecName "kube-api-access-qsxnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:08:05.469904 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.469767    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz7hf\" (UniqueName: \"kubernetes.io/projected/f47dd702-1136-4b91-b42c-b9153b93fc49-kube-api-access-xz7hf\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:05.469904 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.469810    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsxnh\" (UniqueName: \"kubernetes.io/projected/460d7a26-c5fe-47b0-8a5e-bcc6e318ac44-kube-api-access-qsxnh\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:05.469904 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.469850    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47dd702-1136-4b91-b42c-b9153b93fc49-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:05.469904 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.469864    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460d7a26-c5fe-47b0-8a5e-bcc6e318ac44-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:05.632243 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.632209    2581 generic.go:356] "Generic (PLEG): container finished" podID="460d7a26-c5fe-47b0-8a5e-bcc6e318ac44" containerID="378e57ab8e0e1c8cc5f2be3ad1ab2ea0e0bcf35589325ce0c5a7d56d2b3eaa42" exitCode=0
Sep 16 08:08:05.632424 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.632256    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"
Sep 16 08:08:05.632424 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.632294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp" event={"ID":"460d7a26-c5fe-47b0-8a5e-bcc6e318ac44","Type":"ContainerDied","Data":"378e57ab8e0e1c8cc5f2be3ad1ab2ea0e0bcf35589325ce0c5a7d56d2b3eaa42"}
Sep 16 08:08:05.632424 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.632384    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp" event={"ID":"460d7a26-c5fe-47b0-8a5e-bcc6e318ac44","Type":"ContainerDied","Data":"265cac75027050fd992c3c24dcccfb33ebe2e19f742391fb6fdf959962ad5394"}
Sep 16 08:08:05.632424 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.632413    2581 scope.go:117] "RemoveContainer" containerID="378e57ab8e0e1c8cc5f2be3ad1ab2ea0e0bcf35589325ce0c5a7d56d2b3eaa42"
Sep 16 08:08:05.633739 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.633713    2581 generic.go:356] "Generic (PLEG): container finished" podID="f47dd702-1136-4b91-b42c-b9153b93fc49" containerID="462ac42ee413df3a7a999bfd805a28348f76289da4b8530d0e19f1efd7260aeb" exitCode=0
Sep 16 08:08:05.633855 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.633766    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5" event={"ID":"f47dd702-1136-4b91-b42c-b9153b93fc49","Type":"ContainerDied","Data":"462ac42ee413df3a7a999bfd805a28348f76289da4b8530d0e19f1efd7260aeb"}
Sep 16 08:08:05.633855 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.633771    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"
Sep 16 08:08:05.633855 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.633799    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5" event={"ID":"f47dd702-1136-4b91-b42c-b9153b93fc49","Type":"ContainerDied","Data":"964e46ca650ffd5d5196163ce4d2eb74f3815aeb50ef67a9aa21e1821364f320"}
Sep 16 08:08:05.641108 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.641086    2581 scope.go:117] "RemoveContainer" containerID="378e57ab8e0e1c8cc5f2be3ad1ab2ea0e0bcf35589325ce0c5a7d56d2b3eaa42"
Sep 16 08:08:05.641405 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:08:05.641384    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378e57ab8e0e1c8cc5f2be3ad1ab2ea0e0bcf35589325ce0c5a7d56d2b3eaa42\": container with ID starting with 378e57ab8e0e1c8cc5f2be3ad1ab2ea0e0bcf35589325ce0c5a7d56d2b3eaa42 not found: ID does not exist" containerID="378e57ab8e0e1c8cc5f2be3ad1ab2ea0e0bcf35589325ce0c5a7d56d2b3eaa42"
Sep 16 08:08:05.641515 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.641413    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378e57ab8e0e1c8cc5f2be3ad1ab2ea0e0bcf35589325ce0c5a7d56d2b3eaa42"} err="failed to get container status \"378e57ab8e0e1c8cc5f2be3ad1ab2ea0e0bcf35589325ce0c5a7d56d2b3eaa42\": rpc error: code = NotFound desc = could not find container \"378e57ab8e0e1c8cc5f2be3ad1ab2ea0e0bcf35589325ce0c5a7d56d2b3eaa42\": container with ID starting with 378e57ab8e0e1c8cc5f2be3ad1ab2ea0e0bcf35589325ce0c5a7d56d2b3eaa42 not found: ID does not exist"
Sep 16 08:08:05.641515 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.641432    2581 scope.go:117] "RemoveContainer" containerID="462ac42ee413df3a7a999bfd805a28348f76289da4b8530d0e19f1efd7260aeb"
Sep 16 08:08:05.649794 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.649777    2581 scope.go:117] "RemoveContainer" containerID="462ac42ee413df3a7a999bfd805a28348f76289da4b8530d0e19f1efd7260aeb"
Sep 16 08:08:05.650095 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:08:05.650079    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462ac42ee413df3a7a999bfd805a28348f76289da4b8530d0e19f1efd7260aeb\": container with ID starting with 462ac42ee413df3a7a999bfd805a28348f76289da4b8530d0e19f1efd7260aeb not found: ID does not exist" containerID="462ac42ee413df3a7a999bfd805a28348f76289da4b8530d0e19f1efd7260aeb"
Sep 16 08:08:05.650156 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.650102    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462ac42ee413df3a7a999bfd805a28348f76289da4b8530d0e19f1efd7260aeb"} err="failed to get container status \"462ac42ee413df3a7a999bfd805a28348f76289da4b8530d0e19f1efd7260aeb\": rpc error: code = NotFound desc = could not find container \"462ac42ee413df3a7a999bfd805a28348f76289da4b8530d0e19f1efd7260aeb\": container with ID starting with 462ac42ee413df3a7a999bfd805a28348f76289da4b8530d0e19f1efd7260aeb not found: ID does not exist"
Sep 16 08:08:05.658134 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.658114    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"]
Sep 16 08:08:05.661902 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.661881    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5493e8-558b68f76b-tgdrp"]
Sep 16 08:08:05.671940 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.671918    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"]
Sep 16 08:08:05.676057 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.676037    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5493e8-746d5759dc-4pht5"]
Sep 16 08:08:05.684263 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.684243    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="460d7a26-c5fe-47b0-8a5e-bcc6e318ac44" path="/var/lib/kubelet/pods/460d7a26-c5fe-47b0-8a5e-bcc6e318ac44/volumes"
Sep 16 08:08:05.684626 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.684612    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bcd7a2d-625a-4c2e-9722-8f02d98188a6" path="/var/lib/kubelet/pods/8bcd7a2d-625a-4c2e-9722-8f02d98188a6/volumes"
Sep 16 08:08:05.684983 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:05.684971    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f47dd702-1136-4b91-b42c-b9153b93fc49" path="/var/lib/kubelet/pods/f47dd702-1136-4b91-b42c-b9153b93fc49/volumes"
Sep 16 08:08:11.762559 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.762526    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf"]
Sep 16 08:08:11.763060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.762912    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bcd7a2d-625a-4c2e-9722-8f02d98188a6" containerName="revoke-keys"
Sep 16 08:08:11.763060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.762923    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bcd7a2d-625a-4c2e-9722-8f02d98188a6" containerName="revoke-keys"
Sep 16 08:08:11.763060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.762933    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" containerName="istio-proxy"
Sep 16 08:08:11.763060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.762938    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" containerName="istio-proxy"
Sep 16 08:08:11.763060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.762951    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47dd702-1136-4b91-b42c-b9153b93fc49" containerName="payload-processing"
Sep 16 08:08:11.763060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.762957    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47dd702-1136-4b91-b42c-b9153b93fc49" containerName="payload-processing"
Sep 16 08:08:11.763060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.762965    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" containerName="istio-proxy"
Sep 16 08:08:11.763060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.762971    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" containerName="istio-proxy"
Sep 16 08:08:11.763060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.762976    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerName="istio-proxy"
Sep 16 08:08:11.763060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.762981    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="204440a6-7b16-4780-8c29-37d3bd4bdd29" containerName="istio-proxy"
Sep 16 08:08:11.763060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.762994    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="460d7a26-c5fe-47b0-8a5e-bcc6e318ac44" containerName="payload-pre-processing"
Sep 16 08:08:11.763060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.762999    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="460d7a26-c5fe-47b0-8a5e-bcc6e318ac44" containerName="payload-pre-processing"
Sep 16 08:08:11.763060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.763052    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" containerName="istio-proxy"
Sep 16 08:08:11.763060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.763061    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47dd702-1136-4b91-b42c-b9153b93fc49" containerName="payload-processing"
Sep 16 08:08:11.763060 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.763068    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bcd7a2d-625a-4c2e-9722-8f02d98188a6" containerName="revoke-keys"
Sep 16 08:08:11.763515 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.763074    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3fd96c9-b483-44a9-9530-65f20cc8bfe3" containerName="istio-proxy"
Sep 16 08:08:11.763515 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.763081    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="460d7a26-c5fe-47b0-8a5e-bcc6e318ac44" containerName="payload-pre-processing"
Sep 16 08:08:11.767103 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.767085    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf"
Sep 16 08:08:11.769636 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.769578    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\""
Sep 16 08:08:11.769783 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.769728    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 08:08:11.773642 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.773619    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf"]
Sep 16 08:08:11.936696 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:11.936660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd48ac8c-cd24-4d05-9501-4939af325681-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf\" (UID: \"cd48ac8c-cd24-4d05-9501-4939af325681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf"
Sep 16 08:08:12.038102 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:12.037998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd48ac8c-cd24-4d05-9501-4939af325681-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf\" (UID: \"cd48ac8c-cd24-4d05-9501-4939af325681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf"
Sep 16 08:08:12.038802 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:12.038776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd48ac8c-cd24-4d05-9501-4939af325681-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf\" (UID: \"cd48ac8c-cd24-4d05-9501-4939af325681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf"
Sep 16 08:08:12.078964 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:12.078930    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf"
Sep 16 08:08:12.350139 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:12.350112    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf"]
Sep 16 08:08:12.352004 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:08:12.351977    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd48ac8c_cd24_4d05_9501_4939af325681.slice/crio-ad4e4b1c47400f5dbe720409b9f4fa00d05d4f55a9ed0b301968d4ce3d312a1f WatchSource:0}: Error finding container ad4e4b1c47400f5dbe720409b9f4fa00d05d4f55a9ed0b301968d4ce3d312a1f: Status 404 returned error can't find the container with id ad4e4b1c47400f5dbe720409b9f4fa00d05d4f55a9ed0b301968d4ce3d312a1f
Sep 16 08:08:12.672924 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:12.672886    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf" event={"ID":"cd48ac8c-cd24-4d05-9501-4939af325681","Type":"ContainerStarted","Data":"24a8eb439a3c57c37e54a7e1bff96153e9beca907005c92adfe9e2086f0df664"}
Sep 16 08:08:12.673142 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:12.672949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf" event={"ID":"cd48ac8c-cd24-4d05-9501-4939af325681","Type":"ContainerStarted","Data":"ad4e4b1c47400f5dbe720409b9f4fa00d05d4f55a9ed0b301968d4ce3d312a1f"}
Sep 16 08:08:12.690758 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:12.690707    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf" podStartSLOduration=1.6906914899999999 podStartE2EDuration="1.69069149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:08:12.689194902 +0000 UTC m=+2053.615807055" watchObservedRunningTime="2026-09-16 08:08:12.69069149 +0000 UTC m=+2053.617303642"
Sep 16 08:08:13.679555 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:13.679523    2581 generic.go:356] "Generic (PLEG): container finished" podID="cd48ac8c-cd24-4d05-9501-4939af325681" containerID="24a8eb439a3c57c37e54a7e1bff96153e9beca907005c92adfe9e2086f0df664" exitCode=0
Sep 16 08:08:13.683460 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:13.683429    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf" event={"ID":"cd48ac8c-cd24-4d05-9501-4939af325681","Type":"ContainerDied","Data":"24a8eb439a3c57c37e54a7e1bff96153e9beca907005c92adfe9e2086f0df664"}
Sep 16 08:08:14.810734 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:14.810705    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf"
Sep 16 08:08:14.966996 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:14.966906    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd48ac8c-cd24-4d05-9501-4939af325681-maas-api-service-ca\") pod \"cd48ac8c-cd24-4d05-9501-4939af325681\" (UID: \"cd48ac8c-cd24-4d05-9501-4939af325681\") "
Sep 16 08:08:14.967287 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:14.967259    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd48ac8c-cd24-4d05-9501-4939af325681-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd48ac8c-cd24-4d05-9501-4939af325681" (UID: "cd48ac8c-cd24-4d05-9501-4939af325681"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:08:15.067742 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:15.067708    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd48ac8c-cd24-4d05-9501-4939af325681-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:15.689158 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:15.689124    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf"
Sep 16 08:08:15.689158 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:15.689125    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf" event={"ID":"cd48ac8c-cd24-4d05-9501-4939af325681","Type":"ContainerDied","Data":"ad4e4b1c47400f5dbe720409b9f4fa00d05d4f55a9ed0b301968d4ce3d312a1f"}
Sep 16 08:08:15.689379 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:15.689180    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad4e4b1c47400f5dbe720409b9f4fa00d05d4f55a9ed0b301968d4ce3d312a1f"
Sep 16 08:08:22.433298 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:22.433160    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf"]
Sep 16 08:08:22.435612 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:22.435586    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3616cc-6ctxf"]
Sep 16 08:08:23.683991 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:23.683955    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd48ac8c-cd24-4d05-9501-4939af325681" path="/var/lib/kubelet/pods/cd48ac8c-cd24-4d05-9501-4939af325681/volumes"
Sep 16 08:08:28.936673 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:28.936636    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"]
Sep 16 08:08:28.937117 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:28.936975    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g" podUID="514f0b71-05b8-4c88-8baf-2f2667cd22d4" containerName="istio-proxy" containerID="cri-o://d5b7d1c43c0ea9977251230393ca7691c155f3bbeec24b550b8afc381adc4800" gracePeriod=30
Sep 16 08:08:29.323232 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:29.323144    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54"]
Sep 16 08:08:29.323587 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:29.323573    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd48ac8c-cd24-4d05-9501-4939af325681" containerName="revoke-keys"
Sep 16 08:08:29.323631 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:29.323589    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd48ac8c-cd24-4d05-9501-4939af325681" containerName="revoke-keys"
Sep 16 08:08:29.323667 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:29.323660    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd48ac8c-cd24-4d05-9501-4939af325681" containerName="revoke-keys"
Sep 16 08:08:29.326900 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:29.326881    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54"
Sep 16 08:08:29.329376 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:29.329352    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\""
Sep 16 08:08:29.329490 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:29.329352    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 08:08:29.332947 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:29.332924    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54"]
Sep 16 08:08:29.407495 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:29.407460    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d27242-f189-47d5-b0ca-f23493175bdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54\" (UID: \"88d27242-f189-47d5-b0ca-f23493175bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54"
Sep 16 08:08:29.508604 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:29.508561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d27242-f189-47d5-b0ca-f23493175bdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54\" (UID: \"88d27242-f189-47d5-b0ca-f23493175bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54"
Sep 16 08:08:29.509209 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:29.509189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d27242-f189-47d5-b0ca-f23493175bdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54\" (UID: \"88d27242-f189-47d5-b0ca-f23493175bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54"
Sep 16 08:08:29.640024 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:29.639940    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54"
Sep 16 08:08:29.912742 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:29.912713    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54"]
Sep 16 08:08:29.914162 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:08:29.914134    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d27242_f189_47d5_b0ca_f23493175bdf.slice/crio-14803f4da3b9ce9ba32d466464c1ceed0808b722d26dadbd51861d88df664087 WatchSource:0}: Error finding container 14803f4da3b9ce9ba32d466464c1ceed0808b722d26dadbd51861d88df664087: Status 404 returned error can't find the container with id 14803f4da3b9ce9ba32d466464c1ceed0808b722d26dadbd51861d88df664087
Sep 16 08:08:30.766109 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:30.766077    2581 generic.go:356] "Generic (PLEG): container finished" podID="88d27242-f189-47d5-b0ca-f23493175bdf" containerID="10d2dbe6c383afd9030f8781c0a390e392acdb9aa59160957ccdc238fffbb920" exitCode=0
Sep 16 08:08:30.766525 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:30.766160    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54" event={"ID":"88d27242-f189-47d5-b0ca-f23493175bdf","Type":"ContainerDied","Data":"10d2dbe6c383afd9030f8781c0a390e392acdb9aa59160957ccdc238fffbb920"}
Sep 16 08:08:30.766525 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:30.766228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54" event={"ID":"88d27242-f189-47d5-b0ca-f23493175bdf","Type":"ContainerStarted","Data":"14803f4da3b9ce9ba32d466464c1ceed0808b722d26dadbd51861d88df664087"}
Sep 16 08:08:31.905002 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:31.904971    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54"
Sep 16 08:08:32.033074 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:32.032974    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d27242-f189-47d5-b0ca-f23493175bdf-maas-api-service-ca\") pod \"88d27242-f189-47d5-b0ca-f23493175bdf\" (UID: \"88d27242-f189-47d5-b0ca-f23493175bdf\") "
Sep 16 08:08:32.033332 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:32.033309    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d27242-f189-47d5-b0ca-f23493175bdf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88d27242-f189-47d5-b0ca-f23493175bdf" (UID: "88d27242-f189-47d5-b0ca-f23493175bdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:08:32.133739 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:32.133708    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d27242-f189-47d5-b0ca-f23493175bdf-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:32.778978 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:32.778945    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54"
Sep 16 08:08:32.778978 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:32.778961    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54" event={"ID":"88d27242-f189-47d5-b0ca-f23493175bdf","Type":"ContainerDied","Data":"14803f4da3b9ce9ba32d466464c1ceed0808b722d26dadbd51861d88df664087"}
Sep 16 08:08:32.779201 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:32.779009    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14803f4da3b9ce9ba32d466464c1ceed0808b722d26dadbd51861d88df664087"
Sep 16 08:08:34.182775 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.182750    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 08:08:34.254643 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.254551    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-podinfo\") pod \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") "
Sep 16 08:08:34.254643 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.254612    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp6r6\" (UniqueName: \"kubernetes.io/projected/514f0b71-05b8-4c88-8baf-2f2667cd22d4-kube-api-access-bp6r6\") pod \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") "
Sep 16 08:08:34.254892 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.254648    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-data\") pod \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") "
Sep 16 08:08:34.254892 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.254713    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-workload-certs\") pod \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") "
Sep 16 08:08:34.254892 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.254759    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-envoy\") pod \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") "
Sep 16 08:08:34.254892 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.254782    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-token\") pod \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") "
Sep 16 08:08:34.254892 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.254805    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-workload-socket\") pod \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") "
Sep 16 08:08:34.254892 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.254875    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-credential-socket\") pod \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") "
Sep 16 08:08:34.255238 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.254943    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istiod-ca-cert\") pod \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\" (UID: \"514f0b71-05b8-4c88-8baf-2f2667cd22d4\") "
Sep 16 08:08:34.255238 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.254981    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "514f0b71-05b8-4c88-8baf-2f2667cd22d4" (UID: "514f0b71-05b8-4c88-8baf-2f2667cd22d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:08:34.255238 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.255073    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "514f0b71-05b8-4c88-8baf-2f2667cd22d4" (UID: "514f0b71-05b8-4c88-8baf-2f2667cd22d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:08:34.255398 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.255274    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "514f0b71-05b8-4c88-8baf-2f2667cd22d4" (UID: "514f0b71-05b8-4c88-8baf-2f2667cd22d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:08:34.255398 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.255333    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:34.255398 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.255355    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:34.255398 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.255358    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "514f0b71-05b8-4c88-8baf-2f2667cd22d4" (UID: "514f0b71-05b8-4c88-8baf-2f2667cd22d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:08:34.255398 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.255380    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "514f0b71-05b8-4c88-8baf-2f2667cd22d4" (UID: "514f0b71-05b8-4c88-8baf-2f2667cd22d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:08:34.257022 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.256994    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "514f0b71-05b8-4c88-8baf-2f2667cd22d4" (UID: "514f0b71-05b8-4c88-8baf-2f2667cd22d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:08:34.257022 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.257000    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514f0b71-05b8-4c88-8baf-2f2667cd22d4-kube-api-access-bp6r6" (OuterVolumeSpecName: "kube-api-access-bp6r6") pod "514f0b71-05b8-4c88-8baf-2f2667cd22d4" (UID: "514f0b71-05b8-4c88-8baf-2f2667cd22d4"). InnerVolumeSpecName "kube-api-access-bp6r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:08:34.257214 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.257087    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "514f0b71-05b8-4c88-8baf-2f2667cd22d4" (UID: "514f0b71-05b8-4c88-8baf-2f2667cd22d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:08:34.257214 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.257157    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "514f0b71-05b8-4c88-8baf-2f2667cd22d4" (UID: "514f0b71-05b8-4c88-8baf-2f2667cd22d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 08:08:34.356492 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.356453    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:34.356492 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.356485    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:34.356492 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.356495    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp6r6\" (UniqueName: \"kubernetes.io/projected/514f0b71-05b8-4c88-8baf-2f2667cd22d4-kube-api-access-bp6r6\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:34.356736 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.356504    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:34.356736 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.356514    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/514f0b71-05b8-4c88-8baf-2f2667cd22d4-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:34.356736 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.356522    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:34.356736 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.356531    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/514f0b71-05b8-4c88-8baf-2f2667cd22d4-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:34.790987 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.790951    2581 generic.go:356] "Generic (PLEG): container finished" podID="514f0b71-05b8-4c88-8baf-2f2667cd22d4" containerID="d5b7d1c43c0ea9977251230393ca7691c155f3bbeec24b550b8afc381adc4800" exitCode=0
Sep 16 08:08:34.791174 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.791000    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"
Sep 16 08:08:34.791174 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.791035    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g" event={"ID":"514f0b71-05b8-4c88-8baf-2f2667cd22d4","Type":"ContainerDied","Data":"d5b7d1c43c0ea9977251230393ca7691c155f3bbeec24b550b8afc381adc4800"}
Sep 16 08:08:34.791174 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.791091    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g" event={"ID":"514f0b71-05b8-4c88-8baf-2f2667cd22d4","Type":"ContainerDied","Data":"88d614a93ea6043fe0e1fcd4aca7b38d2a2dfd33dbae3f412bb4179333fbb52b"}
Sep 16 08:08:34.791174 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.791126    2581 scope.go:117] "RemoveContainer" containerID="d5b7d1c43c0ea9977251230393ca7691c155f3bbeec24b550b8afc381adc4800"
Sep 16 08:08:34.800692 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.800675    2581 scope.go:117] "RemoveContainer" containerID="8c1ded9a21e07440f49404d8c434fdb02a34d390c4dc466f4ac963f0945f24f7"
Sep 16 08:08:34.809273 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.809251    2581 scope.go:117] "RemoveContainer" containerID="d5b7d1c43c0ea9977251230393ca7691c155f3bbeec24b550b8afc381adc4800"
Sep 16 08:08:34.809584 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:08:34.809565    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b7d1c43c0ea9977251230393ca7691c155f3bbeec24b550b8afc381adc4800\": container with ID starting with d5b7d1c43c0ea9977251230393ca7691c155f3bbeec24b550b8afc381adc4800 not found: ID does not exist" containerID="d5b7d1c43c0ea9977251230393ca7691c155f3bbeec24b550b8afc381adc4800"
Sep 16 08:08:34.809627 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.809595    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b7d1c43c0ea9977251230393ca7691c155f3bbeec24b550b8afc381adc4800"} err="failed to get container status \"d5b7d1c43c0ea9977251230393ca7691c155f3bbeec24b550b8afc381adc4800\": rpc error: code = NotFound desc = could not find container \"d5b7d1c43c0ea9977251230393ca7691c155f3bbeec24b550b8afc381adc4800\": container with ID starting with d5b7d1c43c0ea9977251230393ca7691c155f3bbeec24b550b8afc381adc4800 not found: ID does not exist"
Sep 16 08:08:34.809627 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.809614    2581 scope.go:117] "RemoveContainer" containerID="8c1ded9a21e07440f49404d8c434fdb02a34d390c4dc466f4ac963f0945f24f7"
Sep 16 08:08:34.809908 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:08:34.809890    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1ded9a21e07440f49404d8c434fdb02a34d390c4dc466f4ac963f0945f24f7\": container with ID starting with 8c1ded9a21e07440f49404d8c434fdb02a34d390c4dc466f4ac963f0945f24f7 not found: ID does not exist" containerID="8c1ded9a21e07440f49404d8c434fdb02a34d390c4dc466f4ac963f0945f24f7"
Sep 16 08:08:34.809993 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.809914    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1ded9a21e07440f49404d8c434fdb02a34d390c4dc466f4ac963f0945f24f7"} err="failed to get container status \"8c1ded9a21e07440f49404d8c434fdb02a34d390c4dc466f4ac963f0945f24f7\": rpc error: code = NotFound desc = could not find container \"8c1ded9a21e07440f49404d8c434fdb02a34d390c4dc466f4ac963f0945f24f7\": container with ID starting with 8c1ded9a21e07440f49404d8c434fdb02a34d390c4dc466f4ac963f0945f24f7 not found: ID does not exist"
Sep 16 08:08:34.813420 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.813398    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"]
Sep 16 08:08:34.818238 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:34.818217    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3616cc-openshift-default-5f6bf4648-4zn5g"]
Sep 16 08:08:35.683960 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:35.683922    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="514f0b71-05b8-4c88-8baf-2f2667cd22d4" path="/var/lib/kubelet/pods/514f0b71-05b8-4c88-8baf-2f2667cd22d4/volumes"
Sep 16 08:08:39.874963 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:39.874925    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e9fc8c-29825760-5fn54"]
Sep 16 08:08:39.877793 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:39.877762    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e9fc8c-29825760-5fn54"]
Sep 16 08:08:39.907466 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:39.906887    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"]
Sep 16 08:08:39.907466 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:39.907133    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p" podUID="f499f35e-d8a6-4f67-b3f9-f9377e6d96a3" containerName="payload-pre-processing" containerID="cri-o://26fafa86cff821900ebf246896d90f8ee8fca1112b199da8b096b5c0ee44d233" gracePeriod=30
Sep 16 08:08:40.062358 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.062317    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54"]
Sep 16 08:08:40.064753 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.064722    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e9fc8c-qfk54"]
Sep 16 08:08:40.164076 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.164051    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"
Sep 16 08:08:40.312735 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.312693    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm9dn\" (UniqueName: \"kubernetes.io/projected/f499f35e-d8a6-4f67-b3f9-f9377e6d96a3-kube-api-access-rm9dn\") pod \"f499f35e-d8a6-4f67-b3f9-f9377e6d96a3\" (UID: \"f499f35e-d8a6-4f67-b3f9-f9377e6d96a3\") "
Sep 16 08:08:40.312961 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.312846    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f499f35e-d8a6-4f67-b3f9-f9377e6d96a3-plugins-config-volume\") pod \"f499f35e-d8a6-4f67-b3f9-f9377e6d96a3\" (UID: \"f499f35e-d8a6-4f67-b3f9-f9377e6d96a3\") "
Sep 16 08:08:40.313248 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.313225    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f499f35e-d8a6-4f67-b3f9-f9377e6d96a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f499f35e-d8a6-4f67-b3f9-f9377e6d96a3" (UID: "f499f35e-d8a6-4f67-b3f9-f9377e6d96a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:08:40.314778 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.314748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f499f35e-d8a6-4f67-b3f9-f9377e6d96a3-kube-api-access-rm9dn" (OuterVolumeSpecName: "kube-api-access-rm9dn") pod "f499f35e-d8a6-4f67-b3f9-f9377e6d96a3" (UID: "f499f35e-d8a6-4f67-b3f9-f9377e6d96a3"). InnerVolumeSpecName "kube-api-access-rm9dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:08:40.413536 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.413498    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm9dn\" (UniqueName: \"kubernetes.io/projected/f499f35e-d8a6-4f67-b3f9-f9377e6d96a3-kube-api-access-rm9dn\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:40.413536 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.413534    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f499f35e-d8a6-4f67-b3f9-f9377e6d96a3-plugins-config-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:08:40.827629 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.827518    2581 generic.go:356] "Generic (PLEG): container finished" podID="f499f35e-d8a6-4f67-b3f9-f9377e6d96a3" containerID="26fafa86cff821900ebf246896d90f8ee8fca1112b199da8b096b5c0ee44d233" exitCode=0
Sep 16 08:08:40.827629 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.827586    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"
Sep 16 08:08:40.827629 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.827602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p" event={"ID":"f499f35e-d8a6-4f67-b3f9-f9377e6d96a3","Type":"ContainerDied","Data":"26fafa86cff821900ebf246896d90f8ee8fca1112b199da8b096b5c0ee44d233"}
Sep 16 08:08:40.828020 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.827672    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p" event={"ID":"f499f35e-d8a6-4f67-b3f9-f9377e6d96a3","Type":"ContainerDied","Data":"164e1e649f354b812256d510b56ed971037fcbe497deda7fcc0c816b7572c0dc"}
Sep 16 08:08:40.828020 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.827705    2581 scope.go:117] "RemoveContainer" containerID="26fafa86cff821900ebf246896d90f8ee8fca1112b199da8b096b5c0ee44d233"
Sep 16 08:08:40.839028 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.838994    2581 scope.go:117] "RemoveContainer" containerID="26fafa86cff821900ebf246896d90f8ee8fca1112b199da8b096b5c0ee44d233"
Sep 16 08:08:40.839462 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:08:40.839440    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26fafa86cff821900ebf246896d90f8ee8fca1112b199da8b096b5c0ee44d233\": container with ID starting with 26fafa86cff821900ebf246896d90f8ee8fca1112b199da8b096b5c0ee44d233 not found: ID does not exist" containerID="26fafa86cff821900ebf246896d90f8ee8fca1112b199da8b096b5c0ee44d233"
Sep 16 08:08:40.839523 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.839476    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26fafa86cff821900ebf246896d90f8ee8fca1112b199da8b096b5c0ee44d233"} err="failed to get container status \"26fafa86cff821900ebf246896d90f8ee8fca1112b199da8b096b5c0ee44d233\": rpc error: code = NotFound desc = could not find container \"26fafa86cff821900ebf246896d90f8ee8fca1112b199da8b096b5c0ee44d233\": container with ID starting with 26fafa86cff821900ebf246896d90f8ee8fca1112b199da8b096b5c0ee44d233 not found: ID does not exist"
Sep 16 08:08:40.850357 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.850319    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"]
Sep 16 08:08:40.853704 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:40.853677    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e9fc8c-64f484c9db-gqj5p"]
Sep 16 08:08:41.683581 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:41.683546    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d27242-f189-47d5-b0ca-f23493175bdf" path="/var/lib/kubelet/pods/88d27242-f189-47d5-b0ca-f23493175bdf/volumes"
Sep 16 08:08:41.683964 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:41.683855    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3b48632-21db-4663-aae0-5e4cb613a35a" path="/var/lib/kubelet/pods/b3b48632-21db-4663-aae0-5e4cb613a35a/volumes"
Sep 16 08:08:41.684153 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:41.684139    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f499f35e-d8a6-4f67-b3f9-f9377e6d96a3" path="/var/lib/kubelet/pods/f499f35e-d8a6-4f67-b3f9-f9377e6d96a3/volumes"
Sep 16 08:08:59.693285 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:59.693258    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 08:08:59.698750 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:08:59.698726    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 08:09:00.047394 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:09:00.047300    2581 scope.go:117] "RemoveContainer" containerID="0c2a9885e955bac95d40a4d057db14d5a1f770549a398fc66acd102c013cfcda"
Sep 16 08:13:00.113306 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:00.113227    2581 scope.go:117] "RemoveContainer" containerID="3eae87f0750400df4d4b13343ebaac63dc08e5899d8f53f98a7ea4c1d6a598cc"
Sep 16 08:13:39.663261 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.663218    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"]
Sep 16 08:13:39.663745 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.663613    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f499f35e-d8a6-4f67-b3f9-f9377e6d96a3" containerName="payload-pre-processing"
Sep 16 08:13:39.663745 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.663626    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f499f35e-d8a6-4f67-b3f9-f9377e6d96a3" containerName="payload-pre-processing"
Sep 16 08:13:39.663745 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.663643    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514f0b71-05b8-4c88-8baf-2f2667cd22d4" containerName="istio-proxy"
Sep 16 08:13:39.663745 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.663649    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="514f0b71-05b8-4c88-8baf-2f2667cd22d4" containerName="istio-proxy"
Sep 16 08:13:39.663745 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.663661    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d27242-f189-47d5-b0ca-f23493175bdf" containerName="revoke-keys"
Sep 16 08:13:39.663745 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.663667    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d27242-f189-47d5-b0ca-f23493175bdf" containerName="revoke-keys"
Sep 16 08:13:39.663745 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.663678    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="514f0b71-05b8-4c88-8baf-2f2667cd22d4" containerName="istio-proxy"
Sep 16 08:13:39.663745 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.663684    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="514f0b71-05b8-4c88-8baf-2f2667cd22d4" containerName="istio-proxy"
Sep 16 08:13:39.663745 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.663734    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f499f35e-d8a6-4f67-b3f9-f9377e6d96a3" containerName="payload-pre-processing"
Sep 16 08:13:39.663745 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.663742    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="514f0b71-05b8-4c88-8baf-2f2667cd22d4" containerName="istio-proxy"
Sep 16 08:13:39.663745 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.663749    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="514f0b71-05b8-4c88-8baf-2f2667cd22d4" containerName="istio-proxy"
Sep 16 08:13:39.664232 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.663756    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d27242-f189-47d5-b0ca-f23493175bdf" containerName="revoke-keys"
Sep 16 08:13:39.666807 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.666787    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.669333 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.669312    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c5501a5b-kserve-self-signed-certs\""
Sep 16 08:13:39.677359 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.677337    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"]
Sep 16 08:13:39.750182 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.750147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-dshm\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.750372 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.750190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d7ab1d-5218-4f55-aace-47351a56ecbf-tls-certs\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.750372 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.750215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq6hs\" (UniqueName: \"kubernetes.io/projected/b0d7ab1d-5218-4f55-aace-47351a56ecbf-kube-api-access-kq6hs\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.750372 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.750301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-model-cache\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.750372 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.750351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-tmp-dir\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.750507 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.750417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-home\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.851159 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.851119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-dshm\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.851366 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.851175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d7ab1d-5218-4f55-aace-47351a56ecbf-tls-certs\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.851366 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.851208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq6hs\" (UniqueName: \"kubernetes.io/projected/b0d7ab1d-5218-4f55-aace-47351a56ecbf-kube-api-access-kq6hs\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.851366 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.851245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-model-cache\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.851366 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.851272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-tmp-dir\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.851366 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.851344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-home\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.851653 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.851632    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-model-cache\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.851719 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.851662    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-tmp-dir\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.851780 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.851755    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-home\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.853447 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.853429    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-dshm\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.853722 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.853705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d7ab1d-5218-4f55-aace-47351a56ecbf-tls-certs\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.859067 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.859042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq6hs\" (UniqueName: \"kubernetes.io/projected/b0d7ab1d-5218-4f55-aace-47351a56ecbf-kube-api-access-kq6hs\") pod \"e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") " pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:39.978274 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:39.978182    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:40.250094 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:40.250014    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"]
Sep 16 08:13:40.253310 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:13:40.253279    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0d7ab1d_5218_4f55_aace_47351a56ecbf.slice/crio-3d601a5fbdcd300c63310544f45df52893e5780c1e9640abc640b3b7f139bd09 WatchSource:0}: Error finding container 3d601a5fbdcd300c63310544f45df52893e5780c1e9640abc640b3b7f139bd09: Status 404 returned error can't find the container with id 3d601a5fbdcd300c63310544f45df52893e5780c1e9640abc640b3b7f139bd09
Sep 16 08:13:40.295090 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:40.295054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f" event={"ID":"b0d7ab1d-5218-4f55-aace-47351a56ecbf","Type":"ContainerStarted","Data":"3d601a5fbdcd300c63310544f45df52893e5780c1e9640abc640b3b7f139bd09"}
Sep 16 08:13:41.301624 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:41.301582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f" event={"ID":"b0d7ab1d-5218-4f55-aace-47351a56ecbf","Type":"ContainerStarted","Data":"fb2869bcd9d1ab2073ba9989efe9965d898323d23f41bb825c20eea6196a027b"}
Sep 16 08:13:41.318481 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:41.318416    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f" podStartSLOduration=2.31839385 podStartE2EDuration="2.31839385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:13:41.317507814 +0000 UTC m=+2382.244119967" watchObservedRunningTime="2026-09-16 08:13:41.31839385 +0000 UTC m=+2382.245006004"
Sep 16 08:13:49.979326 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:49.979269    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:49.979846 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:49.979409    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:49.991973 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:49.991945    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:50.356684 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:50.356594    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:13:59.724828 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:59.724789    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 08:13:59.729453 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:13:59.729432    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 08:14:00.132118 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:00.132023    2581 scope.go:117] "RemoveContainer" containerID="95c958f9e7ecb3f98a860e07bdb176a468b358c7a0aa352f0f6737603ae39f50"
Sep 16 08:14:23.328285 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:23.328181    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"]
Sep 16 08:14:38.347722 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.347654    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f" podUID="b0d7ab1d-5218-4f55-aace-47351a56ecbf" containerName="main" containerID="cri-o://fb2869bcd9d1ab2073ba9989efe9965d898323d23f41bb825c20eea6196a027b" gracePeriod=45
Sep 16 08:14:38.582563 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.582531    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0d7ab1d-5218-4f55-aace-47351a56ecbf" containerID="fb2869bcd9d1ab2073ba9989efe9965d898323d23f41bb825c20eea6196a027b" exitCode=0
Sep 16 08:14:38.582695 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.582608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f" event={"ID":"b0d7ab1d-5218-4f55-aace-47351a56ecbf","Type":"ContainerDied","Data":"fb2869bcd9d1ab2073ba9989efe9965d898323d23f41bb825c20eea6196a027b"}
Sep 16 08:14:38.582695 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.582650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f" event={"ID":"b0d7ab1d-5218-4f55-aace-47351a56ecbf","Type":"ContainerDied","Data":"3d601a5fbdcd300c63310544f45df52893e5780c1e9640abc640b3b7f139bd09"}
Sep 16 08:14:38.582695 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.582661    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d601a5fbdcd300c63310544f45df52893e5780c1e9640abc640b3b7f139bd09"
Sep 16 08:14:38.593084 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.593058    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:14:38.614663 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.614588    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kq6hs\" (UniqueName: \"kubernetes.io/projected/b0d7ab1d-5218-4f55-aace-47351a56ecbf-kube-api-access-kq6hs\") pod \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") "
Sep 16 08:14:38.614663 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.614633    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-model-cache\") pod \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") "
Sep 16 08:14:38.614886 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.614672    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-home\") pod \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") "
Sep 16 08:14:38.614886 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.614722    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-tmp-dir\") pod \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") "
Sep 16 08:14:38.614886 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.614765    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d7ab1d-5218-4f55-aace-47351a56ecbf-tls-certs\") pod \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") "
Sep 16 08:14:38.614886 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.614790    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-dshm\") pod \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\" (UID: \"b0d7ab1d-5218-4f55-aace-47351a56ecbf\") "
Sep 16 08:14:38.615119 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.614934    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0d7ab1d-5218-4f55-aace-47351a56ecbf" (UID: "b0d7ab1d-5218-4f55-aace-47351a56ecbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:14:38.615119 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.615012    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-home" (OuterVolumeSpecName: "home") pod "b0d7ab1d-5218-4f55-aace-47351a56ecbf" (UID: "b0d7ab1d-5218-4f55-aace-47351a56ecbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:14:38.615119 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.615030    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0d7ab1d-5218-4f55-aace-47351a56ecbf" (UID: "b0d7ab1d-5218-4f55-aace-47351a56ecbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:14:38.615284 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.615185    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:14:38.615284 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.615227    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:14:38.615284 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.615244    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:14:38.617028 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.617002    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-dshm" (OuterVolumeSpecName: "dshm") pod "b0d7ab1d-5218-4f55-aace-47351a56ecbf" (UID: "b0d7ab1d-5218-4f55-aace-47351a56ecbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 08:14:38.617148 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.617025    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d7ab1d-5218-4f55-aace-47351a56ecbf-kube-api-access-kq6hs" (OuterVolumeSpecName: "kube-api-access-kq6hs") pod "b0d7ab1d-5218-4f55-aace-47351a56ecbf" (UID: "b0d7ab1d-5218-4f55-aace-47351a56ecbf"). InnerVolumeSpecName "kube-api-access-kq6hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:14:38.617148 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.617002    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d7ab1d-5218-4f55-aace-47351a56ecbf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0d7ab1d-5218-4f55-aace-47351a56ecbf" (UID: "b0d7ab1d-5218-4f55-aace-47351a56ecbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 08:14:38.715856 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.715797    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq6hs\" (UniqueName: \"kubernetes.io/projected/b0d7ab1d-5218-4f55-aace-47351a56ecbf-kube-api-access-kq6hs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:14:38.715856 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.715856    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d7ab1d-5218-4f55-aace-47351a56ecbf-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:14:38.716077 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:38.715873    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d7ab1d-5218-4f55-aace-47351a56ecbf-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:14:39.588410 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:39.588379    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"
Sep 16 08:14:39.609390 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:39.609359    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"]
Sep 16 08:14:39.612950 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:39.612923    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c5501a5b-kserve-756dcd796c-jgw7f"]
Sep 16 08:14:39.683268 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:39.683233    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0d7ab1d-5218-4f55-aace-47351a56ecbf" path="/var/lib/kubelet/pods/b0d7ab1d-5218-4f55-aace-47351a56ecbf/volumes"
Sep 16 08:14:42.430163 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.430125    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"]
Sep 16 08:14:42.430556 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.430488    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0d7ab1d-5218-4f55-aace-47351a56ecbf" containerName="main"
Sep 16 08:14:42.430556 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.430499    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0d7ab1d-5218-4f55-aace-47351a56ecbf" containerName="main"
Sep 16 08:14:42.430634 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.430559    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0d7ab1d-5218-4f55-aace-47351a56ecbf" containerName="main"
Sep 16 08:14:42.433806 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.433783    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:42.437553 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.437530    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 08:14:42.437694 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.437555    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fknxb\""
Sep 16 08:14:42.437694 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.437563    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 08:14:42.437694 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.437617    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 08:14:42.437694 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.437662    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 08:14:42.443017 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.442980    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"]
Sep 16 08:14:42.553525 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.553487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc46ba4-a23c-41e6-b76d-89c4460eac82-maas-api-tls\") pod \"maas-api-66d85f5b4d-w9thr\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:42.553718 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.553552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgvq6\" (UniqueName: \"kubernetes.io/projected/6cc46ba4-a23c-41e6-b76d-89c4460eac82-kube-api-access-tgvq6\") pod \"maas-api-66d85f5b4d-w9thr\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:42.553718 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.553608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc46ba4-a23c-41e6-b76d-89c4460eac82-metrics-certs\") pod \"maas-api-66d85f5b4d-w9thr\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:42.553718 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.553670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc46ba4-a23c-41e6-b76d-89c4460eac82-openshift-service-ca\") pod \"maas-api-66d85f5b4d-w9thr\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:42.655068 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.655026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc46ba4-a23c-41e6-b76d-89c4460eac82-maas-api-tls\") pod \"maas-api-66d85f5b4d-w9thr\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:42.655274 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.655075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgvq6\" (UniqueName: \"kubernetes.io/projected/6cc46ba4-a23c-41e6-b76d-89c4460eac82-kube-api-access-tgvq6\") pod \"maas-api-66d85f5b4d-w9thr\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:42.655274 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.655143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc46ba4-a23c-41e6-b76d-89c4460eac82-metrics-certs\") pod \"maas-api-66d85f5b4d-w9thr\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:42.655400 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.655328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc46ba4-a23c-41e6-b76d-89c4460eac82-openshift-service-ca\") pod \"maas-api-66d85f5b4d-w9thr\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:42.656077 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.656020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc46ba4-a23c-41e6-b76d-89c4460eac82-openshift-service-ca\") pod \"maas-api-66d85f5b4d-w9thr\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:42.658412 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.658383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc46ba4-a23c-41e6-b76d-89c4460eac82-metrics-certs\") pod \"maas-api-66d85f5b4d-w9thr\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:42.658556 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.658383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc46ba4-a23c-41e6-b76d-89c4460eac82-maas-api-tls\") pod \"maas-api-66d85f5b4d-w9thr\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:42.663261 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.663235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgvq6\" (UniqueName: \"kubernetes.io/projected/6cc46ba4-a23c-41e6-b76d-89c4460eac82-kube-api-access-tgvq6\") pod \"maas-api-66d85f5b4d-w9thr\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") " pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:42.745531 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:42.745417    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:43.026680 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:43.026643    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"]
Sep 16 08:14:43.028713 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:14:43.028674    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc46ba4_a23c_41e6_b76d_89c4460eac82.slice/crio-c5cce653de885c0c417ca078b8d7eac1277d3e8a941da5d0e03596b1ffe69c12 WatchSource:0}: Error finding container c5cce653de885c0c417ca078b8d7eac1277d3e8a941da5d0e03596b1ffe69c12: Status 404 returned error can't find the container with id c5cce653de885c0c417ca078b8d7eac1277d3e8a941da5d0e03596b1ffe69c12
Sep 16 08:14:43.030409 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:43.030381    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 08:14:43.607316 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:43.607281    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr" event={"ID":"6cc46ba4-a23c-41e6-b76d-89c4460eac82","Type":"ContainerStarted","Data":"c5cce653de885c0c417ca078b8d7eac1277d3e8a941da5d0e03596b1ffe69c12"}
Sep 16 08:14:45.619237 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:45.619199    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr" event={"ID":"6cc46ba4-a23c-41e6-b76d-89c4460eac82","Type":"ContainerStarted","Data":"ea5383af1babcb96455036a41dc678a2a75e76a0d2886bd484a807efe5db02e0"}
Sep 16 08:14:45.619698 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:45.619336    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:45.638615 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:45.638568    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr" podStartSLOduration=1.9937026599999998 podStartE2EDuration="3.638549088s" totalImagesPullingTime="1.644846428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:14:45.637282488 +0000 UTC m=+2446.563894643" watchObservedRunningTime="2026-09-16 08:14:45.638549088 +0000 UTC m=+2446.565161280"
Sep 16 08:14:49.792189 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:49.792141    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"]
Sep 16 08:14:49.792854 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:49.792364    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr" podUID="6cc46ba4-a23c-41e6-b76d-89c4460eac82" containerName="maas-api" containerID="cri-o://ea5383af1babcb96455036a41dc678a2a75e76a0d2886bd484a807efe5db02e0" gracePeriod=30
Sep 16 08:14:49.798525 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:49.798496    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:50.040265 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.040241    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:50.127238 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.127149    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgvq6\" (UniqueName: \"kubernetes.io/projected/6cc46ba4-a23c-41e6-b76d-89c4460eac82-kube-api-access-tgvq6\") pod \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") "
Sep 16 08:14:50.127238 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.127229    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc46ba4-a23c-41e6-b76d-89c4460eac82-maas-api-tls\") pod \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") "
Sep 16 08:14:50.127446 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.127275    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc46ba4-a23c-41e6-b76d-89c4460eac82-openshift-service-ca\") pod \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") "
Sep 16 08:14:50.127446 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.127314    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc46ba4-a23c-41e6-b76d-89c4460eac82-metrics-certs\") pod \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\" (UID: \"6cc46ba4-a23c-41e6-b76d-89c4460eac82\") "
Sep 16 08:14:50.127648 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.127622    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc46ba4-a23c-41e6-b76d-89c4460eac82-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6cc46ba4-a23c-41e6-b76d-89c4460eac82" (UID: "6cc46ba4-a23c-41e6-b76d-89c4460eac82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 08:14:50.129559 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.129531    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc46ba4-a23c-41e6-b76d-89c4460eac82-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6cc46ba4-a23c-41e6-b76d-89c4460eac82" (UID: "6cc46ba4-a23c-41e6-b76d-89c4460eac82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 08:14:50.129678 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.129557    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc46ba4-a23c-41e6-b76d-89c4460eac82-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6cc46ba4-a23c-41e6-b76d-89c4460eac82" (UID: "6cc46ba4-a23c-41e6-b76d-89c4460eac82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 08:14:50.129678 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.129604    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc46ba4-a23c-41e6-b76d-89c4460eac82-kube-api-access-tgvq6" (OuterVolumeSpecName: "kube-api-access-tgvq6") pod "6cc46ba4-a23c-41e6-b76d-89c4460eac82" (UID: "6cc46ba4-a23c-41e6-b76d-89c4460eac82"). InnerVolumeSpecName "kube-api-access-tgvq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 08:14:50.228726 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.228690    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc46ba4-a23c-41e6-b76d-89c4460eac82-maas-api-tls\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:14:50.228726 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.228720    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc46ba4-a23c-41e6-b76d-89c4460eac82-openshift-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:14:50.228726 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.228729    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc46ba4-a23c-41e6-b76d-89c4460eac82-metrics-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:14:50.228726 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.228738    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgvq6\" (UniqueName: \"kubernetes.io/projected/6cc46ba4-a23c-41e6-b76d-89c4460eac82-kube-api-access-tgvq6\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Sep 16 08:14:50.646632 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.646597    2581 generic.go:356] "Generic (PLEG): container finished" podID="6cc46ba4-a23c-41e6-b76d-89c4460eac82" containerID="ea5383af1babcb96455036a41dc678a2a75e76a0d2886bd484a807efe5db02e0" exitCode=0
Sep 16 08:14:50.646844 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.646650    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"
Sep 16 08:14:50.646844 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.646657    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr" event={"ID":"6cc46ba4-a23c-41e6-b76d-89c4460eac82","Type":"ContainerDied","Data":"ea5383af1babcb96455036a41dc678a2a75e76a0d2886bd484a807efe5db02e0"}
Sep 16 08:14:50.646844 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.646700    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr" event={"ID":"6cc46ba4-a23c-41e6-b76d-89c4460eac82","Type":"ContainerDied","Data":"c5cce653de885c0c417ca078b8d7eac1277d3e8a941da5d0e03596b1ffe69c12"}
Sep 16 08:14:50.646844 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.646722    2581 scope.go:117] "RemoveContainer" containerID="ea5383af1babcb96455036a41dc678a2a75e76a0d2886bd484a807efe5db02e0"
Sep 16 08:14:50.656141 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.656120    2581 scope.go:117] "RemoveContainer" containerID="ea5383af1babcb96455036a41dc678a2a75e76a0d2886bd484a807efe5db02e0"
Sep 16 08:14:50.656433 ip-10-0-139-143 kubenswrapper[2581]: E0916 08:14:50.656413    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea5383af1babcb96455036a41dc678a2a75e76a0d2886bd484a807efe5db02e0\": container with ID starting with ea5383af1babcb96455036a41dc678a2a75e76a0d2886bd484a807efe5db02e0 not found: ID does not exist" containerID="ea5383af1babcb96455036a41dc678a2a75e76a0d2886bd484a807efe5db02e0"
Sep 16 08:14:50.656503 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.656441    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea5383af1babcb96455036a41dc678a2a75e76a0d2886bd484a807efe5db02e0"} err="failed to get container status \"ea5383af1babcb96455036a41dc678a2a75e76a0d2886bd484a807efe5db02e0\": rpc error: code = NotFound desc = could not find container \"ea5383af1babcb96455036a41dc678a2a75e76a0d2886bd484a807efe5db02e0\": container with ID starting with ea5383af1babcb96455036a41dc678a2a75e76a0d2886bd484a807efe5db02e0 not found: ID does not exist"
Sep 16 08:14:50.668152 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.668125    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"]
Sep 16 08:14:50.672756 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:50.672729    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-66d85f5b4d-w9thr"]
Sep 16 08:14:51.683971 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:14:51.683936    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cc46ba4-a23c-41e6-b76d-89c4460eac82" path="/var/lib/kubelet/pods/6cc46ba4-a23c-41e6-b76d-89c4460eac82/volumes"
Sep 16 08:15:00.126780 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:00.126732    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825775-fm28p"]
Sep 16 08:15:00.127350 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:00.127115    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc46ba4-a23c-41e6-b76d-89c4460eac82" containerName="maas-api"
Sep 16 08:15:00.127350 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:00.127126    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc46ba4-a23c-41e6-b76d-89c4460eac82" containerName="maas-api"
Sep 16 08:15:00.127350 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:00.127175    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cc46ba4-a23c-41e6-b76d-89c4460eac82" containerName="maas-api"
Sep 16 08:15:00.131183 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:00.131160    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825775-fm28p"
Sep 16 08:15:00.133488 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:00.133467    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\""
Sep 16 08:15:00.137215 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:00.137178    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825775-fm28p"]
Sep 16 08:15:00.142541 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:00.142518    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825775-fm28p"
Sep 16 08:15:00.152258 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:00.152096    2581 scope.go:117] "RemoveContainer" containerID="10d2dbe6c383afd9030f8781c0a390e392acdb9aa59160957ccdc238fffbb920"
Sep 16 08:15:00.178326 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:00.178295    2581 scope.go:117] "RemoveContainer" containerID="24a8eb439a3c57c37e54a7e1bff96153e9beca907005c92adfe9e2086f0df664"
Sep 16 08:15:00.431366 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:00.431342    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825775-fm28p"]
Sep 16 08:15:00.433408 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:15:00.433377    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd6e7c3_86c4_4935_a24a_248162aaff7b.slice/crio-882dc0b7469d2487a40f1454b9911572254765cd257b37e1e26b79ab72ffba82 WatchSource:0}: Error finding container 882dc0b7469d2487a40f1454b9911572254765cd257b37e1e26b79ab72ffba82: Status 404 returned error can't find the container with id 882dc0b7469d2487a40f1454b9911572254765cd257b37e1e26b79ab72ffba82
Sep 16 08:15:00.694937 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:00.694848    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825775-fm28p" event={"ID":"7dd6e7c3-86c4-4935-a24a-248162aaff7b","Type":"ContainerStarted","Data":"882dc0b7469d2487a40f1454b9911572254765cd257b37e1e26b79ab72ffba82"}
Sep 16 08:15:01.701156 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:01.701114    2581 generic.go:356] "Generic (PLEG): container finished" podID="7dd6e7c3-86c4-4935-a24a-248162aaff7b" containerID="c499fdb54fd74205e961b15a0ffc68ffaca98d46cce454d25200a78cb71f2d77" exitCode=0
Sep 16 08:15:01.701538 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:01.701188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825775-fm28p" event={"ID":"7dd6e7c3-86c4-4935-a24a-248162aaff7b","Type":"ContainerDied","Data":"c499fdb54fd74205e961b15a0ffc68ffaca98d46cce454d25200a78cb71f2d77"}
Sep 16 08:15:02.840542 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:02.840519    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825775-fm28p"
Sep 16 08:15:03.711251 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:03.711224    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825775-fm28p"
Sep 16 08:15:03.711403 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:03.711261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825775-fm28p" event={"ID":"7dd6e7c3-86c4-4935-a24a-248162aaff7b","Type":"ContainerDied","Data":"882dc0b7469d2487a40f1454b9911572254765cd257b37e1e26b79ab72ffba82"}
Sep 16 08:15:03.711403 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:03.711313    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="882dc0b7469d2487a40f1454b9911572254765cd257b37e1e26b79ab72ffba82"
Sep 16 08:15:20.642741 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:20.642712    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xnnll_5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb/manager/0.log"
Sep 16 08:15:20.760692 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:20.760651    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-98nd8_382d4153-d0b5-42e4-8232-5910ee40d6a4/manager/0.log"
Sep 16 08:15:21.089974 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:21.089945    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-cnff7_251f9852-5b84-433f-b98f-6728265a094d/manager/0.log"
Sep 16 08:15:21.447879 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:21.447853    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcb658945-wv6kt_248a1a40-3a78-42ff-969b-cdcd9785711e/copy-manifests/0.log"
Sep 16 08:15:21.460350 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:21.460330    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcb658945-wv6kt_248a1a40-3a78-42ff-969b-cdcd9785711e/manager/0.log"
Sep 16 08:15:22.123425 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:22.123402    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29825760-h2trv_91fbad26-b96a-4501-a82d-9b2d8e7e9c24/cleanup/0.log"
Sep 16 08:15:22.227716 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:22.227690    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29825775-fm28p_7dd6e7c3-86c4-4935-a24a-248162aaff7b/cleanup/0.log"
Sep 16 08:15:23.199072 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:23.199043    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-vzns8_c1c72075-54a4-4918-b7de-b7c690cfb55d/manager/0.log"
Sep 16 08:15:24.103348 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:24.103318    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh_b4e2eab0-2feb-42c1-b4d4-386e95a6e661/istio-proxy/0.log"
Sep 16 08:15:24.214182 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:24.214153    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qwxkf_4b1b26dd-431d-476c-9510-74a9f9369d03/discovery/0.log"
Sep 16 08:15:24.433378 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:24.433353    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b6965477b-hhrj5_de638055-4af0-4f62-9181-de3e456e54fe/kube-auth-proxy/0.log"
Sep 16 08:15:24.547616 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:24.547592    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j494b_7c1ff10d-6e1c-4838-a5c6-ccd8e4459859/istio-proxy/1.log"
Sep 16 08:15:24.662690 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:24.662664    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-svx9j_dd4dc9bf-a10b-40fe-8136-9ba3b40549d3/payload-pre-processing/0.log"
Sep 16 08:15:25.204533 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:25.204505    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-nnkn4_1bbedf33-afb5-4a29-b905-35d1a9ea4023/main/0.log"
Sep 16 08:15:25.443802 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:25.443777    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-9zlkg_12c8f434-9910-4015-abac-e9cd82c1f223/main/0.log"
Sep 16 08:15:25.560986 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:25.560908    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-sv4qz_352a18d0-13ab-4b27-aecf-98506046e069/main/0.log"
Sep 16 08:15:33.997185 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:33.997159    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xlmf2_a73e698d-df57-4277-9435-f7d5bfc3be6f/global-pull-secret-syncer/0.log"
Sep 16 08:15:34.073842 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:34.073805    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fsz9s_76e718e8-9332-436d-b35b-89253c6710a7/konnectivity-agent/0.log"
Sep 16 08:15:34.146334 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:34.146303    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-143.ec2.internal_c1a7e92e9ab0a02d77b710fd18e6783c/haproxy/0.log"
Sep 16 08:15:38.495694 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:38.495661    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-vzns8_c1c72075-54a4-4918-b7de-b7c690cfb55d/manager/0.log"
Sep 16 08:15:40.301619 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:40.301591    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e9760136-8a27-4247-bf74-e95bd537b8f2/alertmanager/0.log"
Sep 16 08:15:40.323125 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:40.323101    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e9760136-8a27-4247-bf74-e95bd537b8f2/config-reloader/0.log"
Sep 16 08:15:40.342999 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:40.342974    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e9760136-8a27-4247-bf74-e95bd537b8f2/kube-rbac-proxy-web/0.log"
Sep 16 08:15:40.364475 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:40.364455    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e9760136-8a27-4247-bf74-e95bd537b8f2/kube-rbac-proxy/0.log"
Sep 16 08:15:40.383481 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:40.383458    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e9760136-8a27-4247-bf74-e95bd537b8f2/kube-rbac-proxy-metric/0.log"
Sep 16 08:15:40.402726 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:40.402706    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e9760136-8a27-4247-bf74-e95bd537b8f2/prom-label-proxy/0.log"
Sep 16 08:15:40.422786 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:40.422765    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e9760136-8a27-4247-bf74-e95bd537b8f2/init-config-reloader/0.log"
Sep 16 08:15:40.594419 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:40.594345    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-2pnzc_f9f23028-3891-44db-a55c-056ea019fba9/monitoring-plugin/0.log"
Sep 16 08:15:40.775619 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:40.775593    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tb859_c906e2cd-06e6-4c74-a699-096dcc52f2b1/node-exporter/0.log"
Sep 16 08:15:40.794676 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:40.794655    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tb859_c906e2cd-06e6-4c74-a699-096dcc52f2b1/kube-rbac-proxy/0.log"
Sep 16 08:15:40.812746 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:40.812728    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tb859_c906e2cd-06e6-4c74-a699-096dcc52f2b1/init-textfile/0.log"
Sep 16 08:15:41.162938 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:41.162910    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5fdc6b46f8-2vpvx_bd361603-bf40-4600-88d1-652c1f84158a/telemeter-client/0.log"
Sep 16 08:15:41.186013 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:41.185987    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5fdc6b46f8-2vpvx_bd361603-bf40-4600-88d1-652c1f84158a/reload/0.log"
Sep 16 08:15:41.209342 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:41.209321    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5fdc6b46f8-2vpvx_bd361603-bf40-4600-88d1-652c1f84158a/kube-rbac-proxy/0.log"
Sep 16 08:15:44.562212 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.562178    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"]
Sep 16 08:15:44.562649 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.562526    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd6e7c3-86c4-4935-a24a-248162aaff7b" containerName="cleanup"
Sep 16 08:15:44.562649 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.562537    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd6e7c3-86c4-4935-a24a-248162aaff7b" containerName="cleanup"
Sep 16 08:15:44.562649 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.562597    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd6e7c3-86c4-4935-a24a-248162aaff7b" containerName="cleanup"
Sep 16 08:15:44.565617 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.565600    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.568051 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.568027    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxrzq\"/\"kube-root-ca.crt\""
Sep 16 08:15:44.568169 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.568150    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zxrzq\"/\"default-dockercfg-nr2xl\""
Sep 16 08:15:44.569013 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.568998    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxrzq\"/\"openshift-service-ca.crt\""
Sep 16 08:15:44.572348 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.572327    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"]
Sep 16 08:15:44.724003 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.723969    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/927a0efe-8b71-4bfc-804a-6a49f28df9c1-podres\") pod \"perf-node-gather-daemonset-hdnvm\" (UID: \"927a0efe-8b71-4bfc-804a-6a49f28df9c1\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.724003 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.724004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbg4d\" (UniqueName: \"kubernetes.io/projected/927a0efe-8b71-4bfc-804a-6a49f28df9c1-kube-api-access-vbg4d\") pod \"perf-node-gather-daemonset-hdnvm\" (UID: \"927a0efe-8b71-4bfc-804a-6a49f28df9c1\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.724219 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.724036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/927a0efe-8b71-4bfc-804a-6a49f28df9c1-sys\") pod \"perf-node-gather-daemonset-hdnvm\" (UID: \"927a0efe-8b71-4bfc-804a-6a49f28df9c1\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.724219 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.724118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/927a0efe-8b71-4bfc-804a-6a49f28df9c1-lib-modules\") pod \"perf-node-gather-daemonset-hdnvm\" (UID: \"927a0efe-8b71-4bfc-804a-6a49f28df9c1\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.724219 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.724159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/927a0efe-8b71-4bfc-804a-6a49f28df9c1-proc\") pod \"perf-node-gather-daemonset-hdnvm\" (UID: \"927a0efe-8b71-4bfc-804a-6a49f28df9c1\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.824949 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.824873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/927a0efe-8b71-4bfc-804a-6a49f28df9c1-podres\") pod \"perf-node-gather-daemonset-hdnvm\" (UID: \"927a0efe-8b71-4bfc-804a-6a49f28df9c1\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.824949 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.824909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbg4d\" (UniqueName: \"kubernetes.io/projected/927a0efe-8b71-4bfc-804a-6a49f28df9c1-kube-api-access-vbg4d\") pod \"perf-node-gather-daemonset-hdnvm\" (UID: \"927a0efe-8b71-4bfc-804a-6a49f28df9c1\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.824949 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.824931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/927a0efe-8b71-4bfc-804a-6a49f28df9c1-sys\") pod \"perf-node-gather-daemonset-hdnvm\" (UID: \"927a0efe-8b71-4bfc-804a-6a49f28df9c1\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.825216 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.825000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/927a0efe-8b71-4bfc-804a-6a49f28df9c1-lib-modules\") pod \"perf-node-gather-daemonset-hdnvm\" (UID: \"927a0efe-8b71-4bfc-804a-6a49f28df9c1\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.825216 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.825030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/927a0efe-8b71-4bfc-804a-6a49f28df9c1-proc\") pod \"perf-node-gather-daemonset-hdnvm\" (UID: \"927a0efe-8b71-4bfc-804a-6a49f28df9c1\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.825216 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.825050    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/927a0efe-8b71-4bfc-804a-6a49f28df9c1-podres\") pod \"perf-node-gather-daemonset-hdnvm\" (UID: \"927a0efe-8b71-4bfc-804a-6a49f28df9c1\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.825216 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.825080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/927a0efe-8b71-4bfc-804a-6a49f28df9c1-sys\") pod \"perf-node-gather-daemonset-hdnvm\" (UID: \"927a0efe-8b71-4bfc-804a-6a49f28df9c1\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.825216 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.825111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/927a0efe-8b71-4bfc-804a-6a49f28df9c1-proc\") pod \"perf-node-gather-daemonset-hdnvm\" (UID: \"927a0efe-8b71-4bfc-804a-6a49f28df9c1\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.825216 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.825156    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/927a0efe-8b71-4bfc-804a-6a49f28df9c1-lib-modules\") pod \"perf-node-gather-daemonset-hdnvm\" (UID: \"927a0efe-8b71-4bfc-804a-6a49f28df9c1\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.834165 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.834138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbg4d\" (UniqueName: \"kubernetes.io/projected/927a0efe-8b71-4bfc-804a-6a49f28df9c1-kube-api-access-vbg4d\") pod \"perf-node-gather-daemonset-hdnvm\" (UID: \"927a0efe-8b71-4bfc-804a-6a49f28df9c1\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:44.876505 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:44.876484    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:45.148171 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:45.148145    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"]
Sep 16 08:15:45.150148 ip-10-0-139-143 kubenswrapper[2581]: W0916 08:15:45.150120    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod927a0efe_8b71_4bfc_804a_6a49f28df9c1.slice/crio-1863ebb1dba075b4d673c5db97e478d28092ebb95627941dc0f2f9593c7d553d WatchSource:0}: Error finding container 1863ebb1dba075b4d673c5db97e478d28092ebb95627941dc0f2f9593c7d553d: Status 404 returned error can't find the container with id 1863ebb1dba075b4d673c5db97e478d28092ebb95627941dc0f2f9593c7d553d
Sep 16 08:15:45.908890 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:45.908858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm" event={"ID":"927a0efe-8b71-4bfc-804a-6a49f28df9c1","Type":"ContainerStarted","Data":"5c920c355a1061e9e09d393d2c3e59ef9dca88eaf0ba9ec376804e5a534ffd94"}
Sep 16 08:15:45.909386 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:45.908906    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm" event={"ID":"927a0efe-8b71-4bfc-804a-6a49f28df9c1","Type":"ContainerStarted","Data":"1863ebb1dba075b4d673c5db97e478d28092ebb95627941dc0f2f9593c7d553d"}
Sep 16 08:15:45.909386 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:45.908963    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:45.924920 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:45.924875    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm" podStartSLOduration=1.924862142 podStartE2EDuration="1.924862142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 08:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:15:45.923501096 +0000 UTC m=+2506.850113251" watchObservedRunningTime="2026-09-16 08:15:45.924862142 +0000 UTC m=+2506.851474294"
Sep 16 08:15:45.993112 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:45.993085    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jltjj_fc0d14f2-f58e-40ee-9110-8ce6a97ea069/dns/0.log"
Sep 16 08:15:46.014287 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:46.014268    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jltjj_fc0d14f2-f58e-40ee-9110-8ce6a97ea069/kube-rbac-proxy/0.log"
Sep 16 08:15:46.098802 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:46.098774    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pw9bz_a1b6df6f-8ab5-4abb-87b9-8947b5b36043/dns-node-resolver/0.log"
Sep 16 08:15:46.681047 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:46.681019    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zrs62_d867daeb-9f34-4224-91e7-65889fc3e051/node-ca/0.log"
Sep 16 08:15:47.555655 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:47.555630    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh_b4e2eab0-2feb-42c1-b4d4-386e95a6e661/istio-proxy/0.log"
Sep 16 08:15:47.958736 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:47.958685    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qwxkf_4b1b26dd-431d-476c-9510-74a9f9369d03/discovery/0.log"
Sep 16 08:15:48.003684 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:48.003659    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b6965477b-hhrj5_de638055-4af0-4f62-9181-de3e456e54fe/kube-auth-proxy/0.log"
Sep 16 08:15:48.030126 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:48.030098    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j494b_7c1ff10d-6e1c-4838-a5c6-ccd8e4459859/istio-proxy/0.log"
Sep 16 08:15:48.032941 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:48.032922    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j494b_7c1ff10d-6e1c-4838-a5c6-ccd8e4459859/istio-proxy/1.log"
Sep 16 08:15:48.063475 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:48.063452    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-svx9j_dd4dc9bf-a10b-40fe-8136-9ba3b40549d3/payload-pre-processing/0.log"
Sep 16 08:15:49.164025 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:49.163989    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vcffj_216fb3f5-f1f8-4739-80bf-8d0254b57a8f/serve-healthcheck-canary/0.log"
Sep 16 08:15:49.746645 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:49.746616    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gqj9q_2c32b025-8552-45d9-90ca-df22eadccbab/kube-rbac-proxy/0.log"
Sep 16 08:15:49.767154 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:49.767135    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gqj9q_2c32b025-8552-45d9-90ca-df22eadccbab/exporter/0.log"
Sep 16 08:15:49.794870 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:49.794849    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gqj9q_2c32b025-8552-45d9-90ca-df22eadccbab/extractor/0.log"
Sep 16 08:15:51.923446 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:51.923419    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-hdnvm"
Sep 16 08:15:52.061345 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:52.061280    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xnnll_5fd49635-f9ef-45f6-b9b9-7ab05df1ddeb/manager/0.log"
Sep 16 08:15:52.299115 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:52.299036    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-98nd8_382d4153-d0b5-42e4-8232-5910ee40d6a4/manager/0.log"
Sep 16 08:15:52.518204 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:52.518176    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-cnff7_251f9852-5b84-433f-b98f-6728265a094d/manager/0.log"
Sep 16 08:15:52.697426 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:52.697391    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcb658945-wv6kt_248a1a40-3a78-42ff-969b-cdcd9785711e/manager/0.log"
Sep 16 08:15:52.714229 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:52.714209    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcb658945-wv6kt_248a1a40-3a78-42ff-969b-cdcd9785711e/copy-manifests/0.log"
Sep 16 08:15:54.802789 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:54.802716    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh_b4e2eab0-2feb-42c1-b4d4-386e95a6e661/istio-proxy/0.log"
Sep 16 08:15:55.208833 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:55.208792    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qwxkf_4b1b26dd-431d-476c-9510-74a9f9369d03/discovery/0.log"
Sep 16 08:15:55.232133 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:55.232105    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b6965477b-hhrj5_de638055-4af0-4f62-9181-de3e456e54fe/kube-auth-proxy/0.log"
Sep 16 08:15:55.250725 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:55.250700    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j494b_7c1ff10d-6e1c-4838-a5c6-ccd8e4459859/istio-proxy/0.log"
Sep 16 08:15:55.253471 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:55.253454    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j494b_7c1ff10d-6e1c-4838-a5c6-ccd8e4459859/istio-proxy/1.log"
Sep 16 08:15:55.274580 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:55.274558    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-svx9j_dd4dc9bf-a10b-40fe-8136-9ba3b40549d3/payload-pre-processing/0.log"
Sep 16 08:15:56.829036 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:56.829005    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-tdjqh_b4e2eab0-2feb-42c1-b4d4-386e95a6e661/istio-proxy/0.log"
Sep 16 08:15:57.252281 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:57.252250    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qwxkf_4b1b26dd-431d-476c-9510-74a9f9369d03/discovery/0.log"
Sep 16 08:15:57.290993 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:57.290958    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b6965477b-hhrj5_de638055-4af0-4f62-9181-de3e456e54fe/kube-auth-proxy/0.log"
Sep 16 08:15:57.310983 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:57.310943    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j494b_7c1ff10d-6e1c-4838-a5c6-ccd8e4459859/istio-proxy/0.log"
Sep 16 08:15:57.316212 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:57.316187    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-j494b_7c1ff10d-6e1c-4838-a5c6-ccd8e4459859/istio-proxy/1.log"
Sep 16 08:15:57.342475 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:15:57.342447    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-svx9j_dd4dc9bf-a10b-40fe-8136-9ba3b40549d3/payload-pre-processing/0.log"
Sep 16 08:16:01.037146 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:16:01.037074    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qvcgx_46de41b3-6907-4352-81cf-1216677e98c7/kube-multus-additional-cni-plugins/0.log"
Sep 16 08:16:01.058144 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:16:01.058116    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qvcgx_46de41b3-6907-4352-81cf-1216677e98c7/egress-router-binary-copy/0.log"
Sep 16 08:16:01.078571 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:16:01.078534    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qvcgx_46de41b3-6907-4352-81cf-1216677e98c7/cni-plugins/0.log"
Sep 16 08:16:01.098343 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:16:01.098317    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qvcgx_46de41b3-6907-4352-81cf-1216677e98c7/bond-cni-plugin/0.log"
Sep 16 08:16:01.117472 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:16:01.117449    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qvcgx_46de41b3-6907-4352-81cf-1216677e98c7/routeoverride-cni/0.log"
Sep 16 08:16:01.134731 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:16:01.134709    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qvcgx_46de41b3-6907-4352-81cf-1216677e98c7/whereabouts-cni-bincopy/0.log"
Sep 16 08:16:01.152784 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:16:01.152760    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qvcgx_46de41b3-6907-4352-81cf-1216677e98c7/whereabouts-cni/0.log"
Sep 16 08:16:01.195288 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:16:01.195263    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-h8wqx_6aeb9bac-7199-48cc-8159-16017f479f7e/kube-multus/0.log"
Sep 16 08:16:01.366919 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:16:01.366840    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hrrnm_3b0fd941-35fa-4dac-a59e-38cbea3386d5/network-metrics-daemon/0.log"
Sep 16 08:16:01.381513 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:16:01.381491    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hrrnm_3b0fd941-35fa-4dac-a59e-38cbea3386d5/kube-rbac-proxy/0.log"
Sep 16 08:16:02.243654 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:16:02.243623    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-controller/0.log"
Sep 16 08:16:02.259603 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:16:02.259575    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/0.log"
Sep 16 08:16:02.259755 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:16:02.259610    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/ovn-acl-logging/1.log"
Sep 16 08:16:02.278764 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:16:02.278736    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/kube-rbac-proxy-node/0.log"
Sep 16 08:16:02.300270 ip-10-0-139-143 kubenswrapper[2581]: I0916 08:16:02.300246    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gsjsp_2dfd2dab-c07b-44bf-b150-7e295cd0794c/kube-rbac-proxy-ovn-metrics/0.log"